Sep 16 13:30:37.520920 ip-10-0-139-23 systemd[1]: Starting Kubernetes Kubelet... Sep 16 13:30:37.923586 ip-10-0-139-23 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 13:30:37.923586 ip-10-0-139-23 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 16 13:30:37.923586 ip-10-0-139-23 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 13:30:37.923586 ip-10-0-139-23 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 16 13:30:37.923586 ip-10-0-139-23 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 13:30:37.925198 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.925124 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 16 13:30:37.931678 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931658 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:30:37.931678 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931674 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:30:37.931678 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931678 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:30:37.931678 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931681 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:30:37.931837 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931685 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:30:37.931837 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931689 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:30:37.931837 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931692 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:30:37.931837 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931695 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:30:37.931837 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931698 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:30:37.931837 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931701 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:30:37.931837 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931703 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:30:37.931837 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931706 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:30:37.931837 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931709 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:30:37.931837 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931712 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:30:37.931837 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931714 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:30:37.931837 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931717 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:30:37.931837 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931720 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:30:37.931837 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931722 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:30:37.931837 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931725 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:30:37.931837 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931727 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:30:37.931837 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931729 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:30:37.931837 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931732 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:30:37.931837 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931734 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:30:37.931837 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931737 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:30:37.932301 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931739 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:30:37.932301 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931742 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:30:37.932301 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931744 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:30:37.932301 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931747 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:30:37.932301 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931749 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:30:37.932301 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931752 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:30:37.932301 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931754 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:30:37.932301 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931760 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:30:37.932301 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931763 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:30:37.932301 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931765 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:30:37.932301 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931769 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:30:37.932301 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931771 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:30:37.932301 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931774 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:30:37.932301 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931777 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:30:37.932301 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931780 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:30:37.932301 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931782 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:30:37.932301 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931785 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:30:37.932301 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931787 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:30:37.932301 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931790 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:30:37.932301 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931792 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:30:37.932796 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931795 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:30:37.932796 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931797 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:30:37.932796 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931799 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:30:37.932796 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931802 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:30:37.932796 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931804 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:30:37.932796 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931806 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:30:37.932796 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931809 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:30:37.932796 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931811 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:30:37.932796 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931813 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:30:37.932796 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931815 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:30:37.932796 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931818 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:30:37.932796 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931820 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:30:37.932796 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931823 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:30:37.932796 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931825 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:30:37.932796 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931827 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:30:37.932796 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931830 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:30:37.932796 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931833 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:30:37.932796 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931836 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:30:37.932796 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931838 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:30:37.932796 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931841 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:30:37.933265 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931843 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:30:37.933265 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931846 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:30:37.933265 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931848 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:30:37.933265 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931851 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:30:37.933265 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931853 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:30:37.933265 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931857 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:30:37.933265 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931859 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:30:37.933265 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931862 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:30:37.933265 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931864 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:30:37.933265 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931867 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:30:37.933265 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931869 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:30:37.933265 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931871 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:30:37.933265 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931875 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:30:37.933265 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931880 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:30:37.933265 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931883 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:30:37.933265 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931886 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:30:37.933265 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931889 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:30:37.933265 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931893 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:30:37.933265 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931896 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:30:37.933265 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931899 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:30:37.933745 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931902 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:30:37.933745 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931904 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:30:37.933745 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931907 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:30:37.933745 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931909 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:30:37.933745 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931912 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:30:37.933745 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931914 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:30:37.933745 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931917 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:30:37.933745 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931919 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:30:37.933745 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931921 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:30:37.933745 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931924 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:30:37.933745 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931927 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:30:37.933745 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931930 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:30:37.933745 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931934 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:30:37.933745 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931938 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:30:37.933745 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.931941 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:30:37.933745 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933611 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:30:37.933745 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933635 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:30:37.933745 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933638 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:30:37.933745 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933641 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:30:37.934197 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933644 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:30:37.934197 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933647 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:30:37.934197 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933650 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:30:37.934197 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933652 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:30:37.934197 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933655 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:30:37.934197 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933657 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:30:37.934197 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933660 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:30:37.934197 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933662 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:30:37.934197 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933665 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:30:37.934197 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933667 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:30:37.934197 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933670 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:30:37.934197 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933672 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:30:37.934197 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933674 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:30:37.934197 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933678 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:30:37.934197 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933680 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:30:37.934197 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933684 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:30:37.934197 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933686 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:30:37.934197 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933689 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:30:37.934197 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933692 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:30:37.934197 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933694 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:30:37.934680 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933697 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:30:37.934680 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933699 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:30:37.934680 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933702 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:30:37.934680 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933704 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:30:37.934680 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933707 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:30:37.934680 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933709 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:30:37.934680 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933712 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:30:37.934680 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933714 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:30:37.934680 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933718 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:30:37.934680 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933723 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:30:37.934680 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933726 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:30:37.934680 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933729 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:30:37.934680 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933732 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:30:37.934680 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933735 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:30:37.934680 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933738 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:30:37.934680 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933740 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:30:37.934680 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933743 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:30:37.934680 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933745 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:30:37.934680 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933748 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:30:37.934680 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933750 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:30:37.935204 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933753 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:30:37.935204 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933756 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:30:37.935204 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933758 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:30:37.935204 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933761 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:30:37.935204 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933764 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:30:37.935204 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933766 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:30:37.935204 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933770 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:30:37.935204 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933773 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:30:37.935204 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933775 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:30:37.935204 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933778 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:30:37.935204 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933780 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:30:37.935204 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933783 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:30:37.935204 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933785 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:30:37.935204 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933788 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:30:37.935204 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933790 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:30:37.935204 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933792 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:30:37.935204 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933795 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:30:37.935204 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933797 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:30:37.935204 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933800 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:30:37.935686 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933804 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:30:37.935686 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933807 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:30:37.935686 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933810 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:30:37.935686 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933813 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:30:37.935686 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933816 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:30:37.935686 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933819 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:30:37.935686 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933822 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:30:37.935686 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933824 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:30:37.935686 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933827 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:30:37.935686 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933829 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:30:37.935686 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933832 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:30:37.935686 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933834 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:30:37.935686 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933836 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:30:37.935686 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933839 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:30:37.935686 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933841 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:30:37.935686 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933843 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:30:37.935686 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933846 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:30:37.935686 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933848 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:30:37.935686 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933851 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:30:37.936162 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933854 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:30:37.936162 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933857 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:30:37.936162 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933860 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:30:37.936162 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933863 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:30:37.936162 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933865 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:30:37.936162 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933868 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:30:37.936162 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933870 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:30:37.936162 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933873 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:30:37.936162 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933875 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:30:37.936162 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933878 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:30:37.936162 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933881 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:30:37.936162 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933883 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:30:37.936162 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933885 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:30:37.936162 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933888 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:30:37.936162 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933890 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:30:37.936162 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933893 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:30:37.936162 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.933895 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:30:37.936162 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934258 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:30:37.936162 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934263 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:30:37.936162 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934267 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:30:37.936648 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934269 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:30:37.936648 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934272 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:30:37.936648 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934275 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:30:37.936648 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934278 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:30:37.936648 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934280 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:30:37.936648 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934283 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:30:37.936648 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934285 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:30:37.936648 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934288 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:30:37.936648 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934290 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:30:37.936648 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934293 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:30:37.936648 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934295 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:30:37.936648 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934298 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:30:37.936648 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934301 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:30:37.936648 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934304 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:30:37.936648 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934307 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:30:37.936648 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934309 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:30:37.936648 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934312 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:30:37.936648 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934314 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:30:37.936648 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934317 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:30:37.936648 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934319 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:30:37.937115 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934321 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:30:37.937115 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934324 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:30:37.937115 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934327 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:30:37.937115 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934331 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:30:37.937115 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934333 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:30:37.937115 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934336 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:30:37.937115 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934339 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:30:37.937115 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934342 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:30:37.937115 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934344 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:30:37.937115 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934347 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:30:37.937115 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934349 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:30:37.937115 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934352 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:30:37.937115 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934354 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:30:37.937115 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934358 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:30:37.937115 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934360 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:30:37.937115 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934363 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:30:37.937115 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934365 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:30:37.937115 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934367 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:30:37.937115 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934370 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:30:37.937591 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934372 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:30:37.937591 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934375 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:30:37.937591 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934377 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:30:37.937591 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934379 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:30:37.937591 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934381 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:30:37.937591 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934384 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:30:37.937591 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934387 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:30:37.937591 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934391 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:30:37.937591 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934393 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:30:37.937591 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934396 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:30:37.937591 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934398 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:30:37.937591 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934401 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:30:37.937591 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934403 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:30:37.937591 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934406 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:30:37.937591 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934408 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:30:37.937591 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934410 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:30:37.937591 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934413 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:30:37.937591 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934415 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:30:37.937591 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934418 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:30:37.937591 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934420 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:30:37.938077 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934423 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:30:37.938077 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934425 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:30:37.938077 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934428 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:30:37.938077 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934430 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:30:37.938077 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934432 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:30:37.938077 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934435 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:30:37.938077 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934437 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:30:37.938077 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934440 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:30:37.938077 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934443 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:30:37.938077 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934445 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:30:37.938077 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934448 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:30:37.938077 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934450 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:30:37.938077 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934453 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:30:37.938077 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934456 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:30:37.938077 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934458 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:30:37.938077 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934461 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:30:37.938077 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934463 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:30:37.938077 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934465 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:30:37.938077 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934468 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:30:37.938516 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934472 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:30:37.938516 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934475 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:30:37.938516 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934478 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:30:37.938516 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934480 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:30:37.938516 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934483 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:30:37.938516 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934485 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:30:37.938516 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934487 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:30:37.938516 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934490 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:30:37.938516 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934493 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:30:37.938516 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934496 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:30:37.938516 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934499 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:30:37.938516 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934502 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:30:37.938516 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934504 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:30:37.938516 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934507 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:30:37.938516 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934509 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:30:37.938516 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934512 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:30:37.938516 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934514 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:30:37.938516 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.934516 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:30:37.938516 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934583 2562 flags.go:64] FLAG: --address="0.0.0.0" Sep 16 13:30:37.939000 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934630 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 16 13:30:37.939000 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934636 2562 flags.go:64] FLAG: --anonymous-auth="true" Sep 16 13:30:37.939000 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934640 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 16 13:30:37.939000 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934644 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 16 13:30:37.939000 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934647 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 16 13:30:37.939000 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934651 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 16 13:30:37.939000 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934658 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 16 13:30:37.939000 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934661 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 16 13:30:37.939000 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934664 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 16 13:30:37.939000 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934667 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 16 13:30:37.939000 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934670 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 16 13:30:37.939000 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934673 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 16 13:30:37.939000 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934676 2562 flags.go:64] FLAG: --cgroup-root="" Sep 16 13:30:37.939000 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934678 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 16 13:30:37.939000 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934681 2562 flags.go:64] FLAG: --client-ca-file="" Sep 16 13:30:37.939000 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934685 2562 flags.go:64] FLAG: --cloud-provider="external" Sep 16 13:30:37.939000 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934688 2562 flags.go:64] FLAG: --cluster-dns="[]" Sep 16 13:30:37.939000 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934692 2562 flags.go:64] FLAG: --cluster-domain="" Sep 16 13:30:37.939000 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934695 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 16 13:30:37.939000 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934698 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 16 13:30:37.939000 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934701 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 16 13:30:37.939000 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934704 2562 flags.go:64] FLAG: --container-log-max-files="5" Sep 16 13:30:37.939000 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934707 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 16 13:30:37.939000 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934710 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934713 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934716 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934719 2562 flags.go:64] FLAG: --contention-profiling="false" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934721 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934724 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934727 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934730 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934733 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934736 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934739 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934741 2562 flags.go:64] FLAG: --enable-load-reader="false" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934744 2562 flags.go:64] FLAG: --enable-server="true" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934747 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934750 2562 flags.go:64] FLAG: --event-burst="100" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934753 2562 flags.go:64] FLAG: --event-qps="50" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934756 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934759 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934762 2562 flags.go:64] FLAG: --eviction-hard="" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934765 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934768 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934770 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934773 2562 flags.go:64] FLAG: --eviction-soft="" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934776 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934779 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 16 13:30:37.939569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934782 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 16 13:30:37.940189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934785 2562 flags.go:64] FLAG: --experimental-mounter-path="" Sep 16 13:30:37.940189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934788 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 16 13:30:37.940189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934791 2562 flags.go:64] FLAG: --fail-swap-on="true" Sep 16 13:30:37.940189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934793 2562 flags.go:64] FLAG: --feature-gates="" Sep 16 13:30:37.940189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934797 2562 flags.go:64] FLAG: --file-check-frequency="20s" Sep 16 13:30:37.940189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934800 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 16 13:30:37.940189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934803 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 16 13:30:37.940189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934805 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 16 13:30:37.940189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934808 2562 flags.go:64] FLAG: --healthz-port="10248" Sep 16 13:30:37.940189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934811 2562 flags.go:64] FLAG: --help="false" Sep 16 13:30:37.940189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934814 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-139-23.ec2.internal" Sep 16 13:30:37.940189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934817 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 16 13:30:37.940189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934820 2562 flags.go:64] FLAG: --http-check-frequency="20s" Sep 16 13:30:37.940189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934823 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 16 13:30:37.940189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934826 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 16 13:30:37.940189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934829 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 16 13:30:37.940189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934832 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 16 13:30:37.940189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934835 2562 flags.go:64] FLAG: --image-service-endpoint="" Sep 16 13:30:37.940189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934837 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 16 13:30:37.940189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934840 2562 flags.go:64] FLAG: --kube-api-burst="100" Sep 16 13:30:37.940189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934843 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 16 13:30:37.940189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934846 2562 flags.go:64] FLAG: --kube-api-qps="50" Sep 16 13:30:37.940189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934850 2562 flags.go:64] FLAG: --kube-reserved="" Sep 16 13:30:37.940189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934853 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934856 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934859 2562 flags.go:64] FLAG: --kubelet-cgroups="" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934862 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934864 2562 flags.go:64] FLAG: --lock-file="" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934867 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934870 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934872 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934877 2562 flags.go:64] FLAG: --log-json-split-stream="false" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934880 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934883 2562 flags.go:64] FLAG: --log-text-split-stream="false" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934886 2562 flags.go:64] FLAG: --logging-format="text" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934888 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934891 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934894 2562 flags.go:64] FLAG: --manifest-url="" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934897 2562 flags.go:64] FLAG: --manifest-url-header="" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934901 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934904 2562 flags.go:64] FLAG: --max-open-files="1000000" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934908 2562 flags.go:64] FLAG: --max-pods="110" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934910 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934913 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934917 2562 flags.go:64] FLAG: --memory-manager-policy="None" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934920 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934923 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934925 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 16 13:30:37.940777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934928 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934935 2562 flags.go:64] FLAG: --node-status-max-images="50" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934938 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934941 2562 flags.go:64] FLAG: --oom-score-adj="-999" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934943 2562 flags.go:64] FLAG: --pod-cidr="" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934946 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934952 2562 flags.go:64] FLAG: --pod-manifest-path="" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934955 2562 flags.go:64] FLAG: --pod-max-pids="-1" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934957 2562 flags.go:64] FLAG: --pods-per-core="0" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934960 2562 flags.go:64] FLAG: --port="10250" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934963 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934966 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b61721b6aa174d0b" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934969 2562 flags.go:64] FLAG: --qos-reserved="" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934972 2562 flags.go:64] FLAG: --read-only-port="10255" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934974 2562 flags.go:64] FLAG: --register-node="true" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934977 2562 flags.go:64] FLAG: --register-with-taints="" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934980 2562 flags.go:64] FLAG: --registry-burst="10" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934987 2562 flags.go:64] FLAG: --registry-qps="5" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934990 2562 flags.go:64] FLAG: --reserved-cpus="" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934993 2562 flags.go:64] FLAG: --reserved-memory="" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934996 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.934999 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935002 2562 flags.go:64] FLAG: --rotate-certificates="false" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935005 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935007 2562 flags.go:64] FLAG: --runonce="false" Sep 16 13:30:37.941872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935010 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935013 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935015 2562 flags.go:64] FLAG: --seccomp-default="false" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935018 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935022 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935024 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935027 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935030 2562 flags.go:64] FLAG: --storage-driver-password="root" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935033 2562 flags.go:64] FLAG: --storage-driver-secure="false" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935035 2562 flags.go:64] FLAG: --storage-driver-table="stats" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935038 2562 flags.go:64] FLAG: --storage-driver-user="root" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935041 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935044 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935048 2562 flags.go:64] FLAG: --system-cgroups="" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935050 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935055 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935058 2562 flags.go:64] FLAG: --tls-cert-file="" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935061 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935065 2562 flags.go:64] FLAG: --tls-min-version="" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935068 2562 flags.go:64] FLAG: --tls-private-key-file="" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935070 2562 flags.go:64] FLAG: --topology-manager-policy="none" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935073 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935076 2562 flags.go:64] FLAG: --topology-manager-scope="container" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935078 2562 flags.go:64] FLAG: --v="2" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935082 2562 flags.go:64] FLAG: --version="false" Sep 16 13:30:37.942933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935086 2562 flags.go:64] FLAG: --vmodule="" Sep 16 13:30:37.943710 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935089 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 16 13:30:37.943710 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935092 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 16 13:30:37.943710 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935222 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:30:37.943710 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935226 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:30:37.943710 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935231 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:30:37.943710 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935234 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:30:37.943710 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935238 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:30:37.943710 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935241 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:30:37.943710 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935244 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:30:37.943710 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935247 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:30:37.943710 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935251 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:30:37.943710 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935254 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:30:37.943710 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935257 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:30:37.943710 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935260 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:30:37.943710 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935262 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:30:37.943710 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935265 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:30:37.943710 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935268 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:30:37.943710 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935270 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:30:37.944266 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935273 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:30:37.944266 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935278 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:30:37.944266 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935280 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:30:37.944266 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935283 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:30:37.944266 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935285 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:30:37.944266 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935288 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:30:37.944266 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935290 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:30:37.944266 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935293 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:30:37.944266 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935295 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:30:37.944266 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935298 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:30:37.944266 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935300 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:30:37.944266 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935303 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:30:37.944266 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935305 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:30:37.944266 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935308 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:30:37.944266 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935310 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:30:37.944266 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935314 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:30:37.944266 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935316 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:30:37.944266 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935319 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:30:37.944266 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935321 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:30:37.944266 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935323 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:30:37.945133 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935326 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:30:37.945133 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935328 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:30:37.945133 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935331 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:30:37.945133 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935333 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:30:37.945133 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935337 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:30:37.945133 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935339 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:30:37.945133 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935341 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:30:37.945133 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935344 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:30:37.945133 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935346 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:30:37.945133 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935349 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:30:37.945133 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935351 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:30:37.945133 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935354 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:30:37.945133 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935356 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:30:37.945133 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935360 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:30:37.945133 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935363 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:30:37.945133 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935372 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:30:37.945133 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935375 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:30:37.945133 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935378 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:30:37.945133 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935380 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:30:37.945133 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935383 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:30:37.945781 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935385 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:30:37.945781 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935387 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:30:37.945781 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935390 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:30:37.945781 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935392 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:30:37.945781 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935395 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:30:37.945781 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935397 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:30:37.945781 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935399 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:30:37.945781 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935403 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:30:37.945781 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935405 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:30:37.945781 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935408 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:30:37.945781 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935410 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:30:37.945781 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935413 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:30:37.945781 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935415 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:30:37.945781 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935417 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:30:37.945781 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935420 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:30:37.945781 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935422 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:30:37.945781 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935426 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:30:37.945781 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935429 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:30:37.945781 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935431 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:30:37.945781 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935434 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:30:37.946582 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935437 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:30:37.946582 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935439 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:30:37.946582 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935442 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:30:37.946582 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935445 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:30:37.946582 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935448 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:30:37.946582 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935453 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:30:37.946582 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935456 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:30:37.946582 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935458 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:30:37.946582 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935461 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:30:37.946582 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935463 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:30:37.946582 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935466 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:30:37.946582 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935468 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:30:37.946582 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935471 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:30:37.946582 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935473 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:30:37.946582 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935475 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:30:37.946582 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935478 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:30:37.946582 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935480 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:30:37.946582 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935483 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:30:37.946582 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935485 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:30:37.946582 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935488 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:30:37.947406 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935494 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:30:37.947406 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935496 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:30:37.947406 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.935499 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:30:37.947406 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.935507 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 13:30:37.947406 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.943154 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 16 13:30:37.947406 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.943173 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 16 13:30:37.947406 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.943972 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 16 13:30:37.947406 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.943985 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 16 13:30:37.947406 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944076 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:30:37.947406 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944086 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:30:37.947406 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944093 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:30:37.947406 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944099 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:30:37.947406 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944103 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:30:37.947406 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944108 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:30:37.947767 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944112 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:30:37.947767 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944117 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:30:37.947767 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944122 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:30:37.947767 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944126 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:30:37.947767 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944133 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:30:37.947767 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944138 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:30:37.947767 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944143 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:30:37.947767 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944149 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:30:37.947767 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944154 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:30:37.947767 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944159 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:30:37.947767 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944164 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:30:37.947767 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944168 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:30:37.947767 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944172 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:30:37.947767 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944177 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:30:37.947767 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944181 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:30:37.947767 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944185 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:30:37.947767 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944189 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:30:37.947767 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944194 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:30:37.947767 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944199 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:30:37.948358 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944203 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:30:37.948358 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944207 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:30:37.948358 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944211 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:30:37.948358 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944215 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:30:37.948358 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944221 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:30:37.948358 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944225 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:30:37.948358 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944229 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:30:37.948358 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944234 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:30:37.948358 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944238 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:30:37.948358 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944242 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:30:37.948358 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944246 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:30:37.948358 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944250 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:30:37.948358 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944254 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:30:37.948358 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944258 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:30:37.948358 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944262 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:30:37.948358 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944265 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:30:37.948358 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944269 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:30:37.948358 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944273 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:30:37.948358 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944277 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:30:37.948358 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944280 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:30:37.948858 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944284 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:30:37.948858 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944288 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:30:37.948858 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944292 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:30:37.948858 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944296 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:30:37.948858 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944300 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:30:37.948858 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944304 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:30:37.948858 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944308 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:30:37.948858 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944312 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:30:37.948858 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944317 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:30:37.948858 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944321 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:30:37.948858 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944324 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:30:37.948858 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944328 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:30:37.948858 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944334 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:30:37.948858 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944339 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:30:37.948858 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944343 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:30:37.948858 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944347 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:30:37.948858 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944352 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:30:37.948858 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944356 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:30:37.948858 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944360 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:30:37.948858 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944364 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:30:37.949370 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944368 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:30:37.949370 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944372 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:30:37.949370 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944375 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:30:37.949370 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944380 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:30:37.949370 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944384 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:30:37.949370 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944388 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:30:37.949370 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944392 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:30:37.949370 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944396 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:30:37.949370 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944400 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:30:37.949370 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944404 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:30:37.949370 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944408 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:30:37.949370 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944412 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:30:37.949370 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944415 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:30:37.949370 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944420 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:30:37.949370 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944424 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:30:37.949370 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944428 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:30:37.949370 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944432 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:30:37.949370 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944436 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:30:37.949370 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944440 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:30:37.949370 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944444 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:30:37.949989 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944448 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:30:37.949989 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944452 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:30:37.949989 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944455 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:30:37.949989 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944459 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:30:37.949989 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944463 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:30:37.949989 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944469 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:30:37.949989 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944472 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:30:37.949989 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944477 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:30:37.949989 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944482 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:30:37.949989 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944486 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:30:37.949989 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944490 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:30:37.949989 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944494 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:30:37.949989 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944498 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:30:37.949989 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944502 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:30:37.949989 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.944510 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 13:30:37.950350 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944733 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 13:30:37.950350 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944742 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 13:30:37.950350 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944747 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 13:30:37.950350 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944751 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 13:30:37.950350 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944758 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 13:30:37.950350 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944764 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 13:30:37.950350 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944768 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 13:30:37.950350 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944773 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 13:30:37.950350 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944777 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 13:30:37.950350 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944781 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 13:30:37.950350 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944786 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 13:30:37.950350 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944790 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 13:30:37.950350 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944794 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 13:30:37.950350 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944798 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 13:30:37.950350 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944803 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 13:30:37.950350 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944806 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 13:30:37.950350 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944811 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 13:30:37.950350 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944815 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 13:30:37.950350 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944819 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 13:30:37.950814 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944822 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 13:30:37.950814 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944827 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 13:30:37.950814 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944833 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 13:30:37.950814 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944838 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 13:30:37.950814 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944843 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 13:30:37.950814 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944848 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 13:30:37.950814 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944853 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 16 13:30:37.950814 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944858 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 13:30:37.950814 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944862 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 13:30:37.950814 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944866 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 13:30:37.950814 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944871 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 13:30:37.950814 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944875 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 13:30:37.950814 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944879 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 13:30:37.950814 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944883 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 13:30:37.950814 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944888 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 13:30:37.950814 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944892 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 13:30:37.950814 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944896 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 13:30:37.950814 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944900 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 13:30:37.950814 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944904 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 13:30:37.951280 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944909 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 13:30:37.951280 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944913 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 13:30:37.951280 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944917 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 13:30:37.951280 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944920 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 13:30:37.951280 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944924 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 13:30:37.951280 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944928 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 13:30:37.951280 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944932 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 13:30:37.951280 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944936 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 13:30:37.951280 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944940 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 13:30:37.951280 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944944 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 13:30:37.951280 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944948 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 13:30:37.951280 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944952 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 13:30:37.951280 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944956 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 13:30:37.951280 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944960 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 13:30:37.951280 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944964 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 13:30:37.951280 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944968 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 13:30:37.951280 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944973 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 13:30:37.951280 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944977 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 13:30:37.951280 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944982 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 13:30:37.951280 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944986 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 13:30:37.951983 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944990 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 13:30:37.951983 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944994 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 13:30:37.951983 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.944998 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 13:30:37.951983 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945002 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 13:30:37.951983 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945006 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 13:30:37.951983 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945010 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 13:30:37.951983 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945014 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 13:30:37.951983 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945018 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 13:30:37.951983 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945022 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 13:30:37.951983 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945026 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 13:30:37.951983 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945029 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 13:30:37.951983 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945033 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 13:30:37.951983 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945039 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 13:30:37.951983 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945044 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 13:30:37.951983 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945048 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 13:30:37.951983 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945052 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 13:30:37.951983 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945056 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 13:30:37.951983 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945060 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 13:30:37.951983 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945064 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 13:30:37.951983 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945068 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 13:30:37.952801 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945072 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 13:30:37.952801 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945076 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 13:30:37.952801 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945080 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 13:30:37.952801 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945084 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 13:30:37.952801 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945088 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 13:30:37.952801 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945092 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 13:30:37.952801 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945096 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 13:30:37.952801 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945100 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 13:30:37.952801 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945104 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 13:30:37.952801 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945108 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 13:30:37.952801 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945112 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 13:30:37.952801 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945125 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 13:30:37.952801 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945129 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 13:30:37.952801 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945133 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 13:30:37.952801 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945136 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 13:30:37.952801 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945140 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 13:30:37.952801 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945144 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 13:30:37.952801 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945147 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 13:30:37.952801 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945151 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 13:30:37.952801 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945155 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 13:30:37.953390 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:37.945159 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 13:30:37.953390 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.945167 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 13:30:37.953390 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.945298 2562 server.go:962] "Client rotation is on, will bootstrap in background" Sep 16 13:30:37.953390 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.947865 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 16 13:30:37.953390 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.948919 2562 server.go:1019] "Starting client certificate rotation" Sep 16 13:30:37.953390 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.949032 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 13:30:37.953390 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.949068 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 13:30:37.953390 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.952213 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 13:30:37.960153 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.960134 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 13:30:37.969926 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.969912 2562 log.go:25] "Validated CRI v1 runtime API" Sep 16 13:30:37.975246 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.975233 2562 log.go:25] "Validated CRI v1 image API" Sep 16 13:30:37.976116 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.976087 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 16 13:30:37.978721 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.978705 2562 fs.go:135] Filesystem UUIDs: map[64f99828-52af-4b51-8918-d05ddfe2d189:/dev/nvme0n1p4 6df2762a-151d-4eef-aad0-d8703deae57a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Sep 16 13:30:37.978792 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.978721 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 16 13:30:37.978835 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.978824 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 13:30:37.983402 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.983298 2562 manager.go:216] Machine: {Timestamp:2026-09-16 13:30:37.982098328 +0000 UTC m=+0.345158027 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3201848 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2442a22be242320599afa608a24e89 SystemUUID:ec2442a2-2be2-4232-0599-afa608a24e89 BootID:528f1c4d-c16d-4d11-97fb-b959f72f29fa Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d8:b3:29:ff Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d8:b3:29:ff Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:4f:6d:d2:a7:2d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 16 13:30:37.983402 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.983400 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 16 13:30:37.983507 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.983467 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 16 13:30:37.984533 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.984515 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 16 13:30:37.984680 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.984537 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-23.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 16 13:30:37.985755 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.985745 2562 topology_manager.go:138] "Creating topology manager with none policy" Sep 16 13:30:37.985786 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.985759 2562 container_manager_linux.go:309] "Creating device plugin manager" Sep 16 13:30:37.985786 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.985766 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 13:30:37.986678 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.986668 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 13:30:37.987653 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.987643 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 16 13:30:37.991412 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.991400 2562 state_mem.go:36] "Initialized new in-memory state store" Sep 16 13:30:37.991516 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.991507 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 16 13:30:37.991546 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.991536 2562 kubelet.go:486] "Attempting to sync node with API server" Sep 16 13:30:37.991546 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.991545 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 16 13:30:37.991603 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.991558 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 16 13:30:37.991603 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.991568 2562 kubelet.go:398] "Adding apiserver pod source" Sep 16 13:30:37.991603 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.991576 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 16 13:30:37.991938 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.991927 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 13:30:37.991999 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.991945 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 13:30:37.993947 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.993931 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 16 13:30:37.996276 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.996256 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 16 13:30:37.996362 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.996298 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 16 13:30:37.997831 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.997817 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 16 13:30:37.997963 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.997837 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 16 13:30:37.997963 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.997845 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 16 13:30:37.997963 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.997852 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 16 13:30:37.997963 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.997864 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 16 13:30:37.997963 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.997871 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 16 13:30:37.997963 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.997889 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 16 13:30:37.997963 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.997896 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 16 13:30:37.997963 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.997910 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 16 13:30:37.997963 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.997917 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 16 13:30:37.997963 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.997929 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 16 13:30:37.997963 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.997936 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 16 13:30:37.999198 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.999187 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 16 13:30:37.999257 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:37.999201 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 16 13:30:38.001935 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.001916 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-23.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 16 13:30:38.002069 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.001972 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-23.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 16 13:30:38.002164 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.002142 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 16 13:30:38.002927 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.002911 2562 server.go:1268] "Started kubelet" Sep 16 13:30:38.003384 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.002968 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 16 13:30:38.003384 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.003040 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 16 13:30:38.003384 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.003130 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 16 13:30:38.003731 ip-10-0-139-23 systemd[1]: Started Kubernetes Kubelet. Sep 16 13:30:38.006834 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.006812 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 16 13:30:38.009267 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.009208 2562 server.go:310] "Adding debug handlers to kubelet server" Sep 16 13:30:38.010324 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.010302 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 16 13:30:38.012589 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.012566 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 16 13:30:38.012589 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.012583 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 16 13:30:38.012748 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.012647 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found" Sep 16 13:30:38.013037 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.012051 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-23.ec2.internal.18d5d048ca7a54b6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-23.ec2.internal,UID:ip-10-0-139-23.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-23.ec2.internal,},FirstTimestamp:2026-09-16 13:30:38.002877622 +0000 UTC m=+0.365937312,LastTimestamp:2026-09-16 13:30:38.002877622 +0000 UTC m=+0.365937312,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-23.ec2.internal,}" Sep 16 13:30:38.013416 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.013234 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 16 13:30:38.013416 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.013249 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 16 13:30:38.013416 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.013247 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 16 13:30:38.013416 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.013319 2562 reconstruct.go:97] "Volume reconstruction finished" Sep 16 13:30:38.013416 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.013326 2562 reconciler.go:29] "Reconciler: start to sync state" Sep 16 13:30:38.013776 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.013754 2562 factory.go:55] Registering systemd factory Sep 16 13:30:38.013776 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.013773 2562 factory.go:223] Registration of the systemd container factory successfully Sep 16 13:30:38.013964 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.013949 2562 factory.go:153] Registering CRI-O factory Sep 16 13:30:38.013964 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.013964 2562 factory.go:223] Registration of the crio container factory successfully Sep 16 13:30:38.014076 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.014052 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 16 13:30:38.014076 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.014072 2562 factory.go:103] Registering Raw factory Sep 16 13:30:38.014146 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.014086 2562 manager.go:1196] Started watching for new ooms in manager Sep 16 13:30:38.014488 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.014476 2562 manager.go:349] Starting recovery of all containers Sep 16 13:30:38.020578 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.020406 2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-23.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 16 13:30:38.020680 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.020641 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 16 13:30:38.021510 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.021469 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 16 13:30:38.024884 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.024867 2562 manager.go:354] Recovery completed Sep 16 13:30:38.026173 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.026152 2562 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Sep 16 13:30:38.028823 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.028813 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:30:38.029983 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.029966 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bvntv" Sep 16 13:30:38.030483 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.030471 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:30:38.030551 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.030499 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:30:38.030551 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.030514 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:30:38.031035 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.031022 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 16 13:30:38.031090 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.031037 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 16 13:30:38.031090 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.031054 2562 state_mem.go:36] "Initialized new in-memory state store" Sep 16 13:30:38.031090 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.031071 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:30:38.032434 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.032416 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:30:38.032524 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.032442 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:30:38.032524 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.032456 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:30:38.032524 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.032453 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-23.ec2.internal.18d5d048cc1f9290 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-23.ec2.internal,UID:ip-10-0-139-23.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-23.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-23.ec2.internal,},FirstTimestamp:2026-09-16 13:30:38.030484112 +0000 UTC m=+0.393543808,LastTimestamp:2026-09-16 13:30:38.030484112 +0000 UTC m=+0.393543808,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-23.ec2.internal,}" Sep 16 13:30:38.033268 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.033256 2562 policy_none.go:49] "None policy: Start" Sep 16 13:30:38.033309 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.033272 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 16 13:30:38.033309 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.033283 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 16 13:30:38.035196 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.035185 2562 policy_none.go:47] "Start" Sep 16 13:30:38.038820 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.038800 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bvntv" Sep 16 13:30:38.043422 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.043357 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-23.ec2.internal.18d5d048cc1fe83c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-23.ec2.internal,UID:ip-10-0-139-23.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-23.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-23.ec2.internal,},FirstTimestamp:2026-09-16 13:30:38.030506044 +0000 UTC m=+0.393565733,LastTimestamp:2026-09-16 13:30:38.030506044 +0000 UTC m=+0.393565733,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-23.ec2.internal,}" Sep 16 13:30:38.080192 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.080177 2562 manager.go:337] "Starting Device Plugin manager" Sep 16 13:30:38.101275 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.080210 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 16 13:30:38.101275 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.080219 2562 server.go:87] "Starting device plugin registration server" Sep 16 13:30:38.101275 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.080497 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 16 13:30:38.101275 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.080511 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 16 13:30:38.101275 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.080611 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 16 13:30:38.101275 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.080734 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 16 13:30:38.101275 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.080744 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 16 13:30:38.101275 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.081164 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 16 13:30:38.101275 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.081209 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-23.ec2.internal\" not found" Sep 16 13:30:38.101275 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.087820 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 16 13:30:38.101275 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.087862 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 16 13:30:38.101275 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.087878 2562 kubelet.go:2448] "Starting kubelet main sync loop" Sep 16 13:30:38.101275 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.087910 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 16 13:30:38.101275 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.091034 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:30:38.181718 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.181638 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:30:38.182472 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.182458 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:30:38.182529 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.182486 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:30:38.182529 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.182496 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:30:38.182529 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.182516 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-23.ec2.internal" Sep 16 13:30:38.188611 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.188581 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-23.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal"] Sep 16 13:30:38.188696 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.188673 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:30:38.190021 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.190006 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:30:38.190096 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.190037 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:30:38.190096 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.190051 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:30:38.192172 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.192158 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:30:38.192280 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.192266 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-23.ec2.internal" Sep 16 13:30:38.192361 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.192299 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:30:38.192361 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.192272 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-23.ec2.internal" Sep 16 13:30:38.192361 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.192318 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-23.ec2.internal\": node \"ip-10-0-139-23.ec2.internal\" not found" Sep 16 13:30:38.192855 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.192833 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:30:38.192855 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.192848 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:30:38.192998 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.192862 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:30:38.192998 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.192870 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:30:38.192998 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.192877 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:30:38.192998 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.192882 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:30:38.194944 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.194928 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal" Sep 16 13:30:38.195024 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.194955 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:30:38.195603 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.195589 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:30:38.195689 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.195642 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:30:38.195689 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.195658 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:30:38.208106 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.208089 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-23.ec2.internal\" not found" node="ip-10-0-139-23.ec2.internal" Sep 16 13:30:38.212207 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.212192 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-23.ec2.internal\" not found" node="ip-10-0-139-23.ec2.internal" Sep 16 13:30:38.228721 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.228700 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found" Sep 16 13:30:38.314707 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.314684 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc88820cac2d990b85f97c81f96e2d49-config\") pod \"kube-apiserver-proxy-ip-10-0-139-23.ec2.internal\" (UID: \"fc88820cac2d990b85f97c81f96e2d49\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-23.ec2.internal" Sep 16 13:30:38.314707 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.314708 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7bf8d3fcfbbc3832a3c68332bac0716-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal\" (UID: \"b7bf8d3fcfbbc3832a3c68332bac0716\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal" Sep 16 13:30:38.314817 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.314722 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7bf8d3fcfbbc3832a3c68332bac0716-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal\" (UID: \"b7bf8d3fcfbbc3832a3c68332bac0716\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal" Sep 16 13:30:38.329161 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.329143 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found" Sep 16 13:30:38.415471 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.415453 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc88820cac2d990b85f97c81f96e2d49-config\") pod \"kube-apiserver-proxy-ip-10-0-139-23.ec2.internal\" (UID: \"fc88820cac2d990b85f97c81f96e2d49\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-23.ec2.internal" Sep 16 13:30:38.415518 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.415475 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7bf8d3fcfbbc3832a3c68332bac0716-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal\" (UID: \"b7bf8d3fcfbbc3832a3c68332bac0716\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal" Sep 16 13:30:38.415518 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.415500 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7bf8d3fcfbbc3832a3c68332bac0716-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal\" (UID: \"b7bf8d3fcfbbc3832a3c68332bac0716\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal" Sep 16 13:30:38.415588 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.415551 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7bf8d3fcfbbc3832a3c68332bac0716-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal\" (UID: \"b7bf8d3fcfbbc3832a3c68332bac0716\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal" Sep 16 13:30:38.415588 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.415563 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7bf8d3fcfbbc3832a3c68332bac0716-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal\" (UID: \"b7bf8d3fcfbbc3832a3c68332bac0716\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal" Sep 16 13:30:38.415588 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.415551 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc88820cac2d990b85f97c81f96e2d49-config\") pod \"kube-apiserver-proxy-ip-10-0-139-23.ec2.internal\" (UID: \"fc88820cac2d990b85f97c81f96e2d49\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-23.ec2.internal" Sep 16 13:30:38.429557 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.429537 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found" Sep 16 13:30:38.511023 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.510977 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-23.ec2.internal" Sep 16 13:30:38.514516 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.514485 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal" Sep 16 13:30:38.529978 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.529960 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found" Sep 16 13:30:38.630534 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.630503 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found" Sep 16 13:30:38.730957 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.730932 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found" Sep 16 13:30:38.831499 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.831452 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found" Sep 16 13:30:38.902107 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.902089 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:30:38.932215 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:38.932189 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found" Sep 16 13:30:38.948281 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.948264 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 16 13:30:38.948396 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.948371 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 13:30:38.948433 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:38.948410 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 13:30:39.033272 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:39.033249 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found" Sep 16 13:30:39.042409 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:39.042387 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 13:25:39 +0000 UTC" deadline="2028-05-01 21:22:51.570157836 +0000 UTC" Sep 16 13:30:39.042455 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:39.042408 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14239h52m12.527752639s" Sep 16 13:30:39.060540 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:39.060526 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 13:30:39.063107 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:39.062846 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientMemory" Sep 16 13:30:39.063107 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:39.062874 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 13:30:39.063107 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:39.062887 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientPID" Sep 16 13:30:39.133994 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:39.133947 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found" Sep 16 13:30:39.185564 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:39.185549 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:30:39.235047 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:39.235021 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found" Sep 16 13:30:39.303554 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:39.303522 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc88820cac2d990b85f97c81f96e2d49.slice/crio-46353344800aea6b4d03c4c3b08e98aca2e0f1c1a9f673f1cc77cc6eb4905609 WatchSource:0}: Error finding container 46353344800aea6b4d03c4c3b08e98aca2e0f1c1a9f673f1cc77cc6eb4905609: Status 404 returned error can't find the container with id 46353344800aea6b4d03c4c3b08e98aca2e0f1c1a9f673f1cc77cc6eb4905609 Sep 16 13:30:39.303788 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:39.303774 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7bf8d3fcfbbc3832a3c68332bac0716.slice/crio-31d49837f9700c4784d13868dbd7f5963cda2fe0349e8fed4dbc8f03ceacb634 WatchSource:0}: Error finding container 31d49837f9700c4784d13868dbd7f5963cda2fe0349e8fed4dbc8f03ceacb634: Status 404 returned error can't find the container with id 31d49837f9700c4784d13868dbd7f5963cda2fe0349e8fed4dbc8f03ceacb634 Sep 16 13:30:39.308227 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:39.308215 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 13:30:39.335107 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:39.335076 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found" Sep 16 13:30:39.396585 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:39.396537 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:30:39.413066 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:39.413051 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal" Sep 16 13:30:39.430235 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:39.430220 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 13:30:39.431551 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:39.431539 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-23.ec2.internal" Sep 16 13:30:39.443024 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:39.443007 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 13:30:41.312339 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.312272 2562 apiserver.go:52] "Watching apiserver" Sep 16 13:30:41.324329 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.324306 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 16 13:30:41.324682 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.324659 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-8zxxf","kube-system/kube-apiserver-proxy-ip-10-0-139-23.ec2.internal","openshift-image-registry/node-ca-mcc87","openshift-multus/multus-additional-cni-plugins-d6bps","openshift-multus/multus-dwt2h","openshift-network-diagnostics/network-check-target-vg66w","openshift-network-operator/iptables-alerter-hwrln","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v","openshift-cluster-node-tuning-operator/tuned-kmpb5","openshift-dns/node-resolver-t9qw9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal","openshift-multus/network-metrics-daemon-hm6lg","openshift-ovn-kubernetes/ovnkube-node-2k2tg","kube-system/global-pull-secret-syncer-5s6pv"] Sep 16 13:30:41.328469 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.328447 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hwrln" Sep 16 13:30:41.330626 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.330611 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.331733 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.331719 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 16 13:30:41.332411 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.332364 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 16 13:30:41.332507 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.332447 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mxmlk\"" Sep 16 13:30:41.332731 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.332716 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mcc87" Sep 16 13:30:41.333372 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.333355 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 16 13:30:41.333479 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.333360 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:30:41.333479 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.333443 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 16 13:30:41.333749 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.333717 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 16 13:30:41.333812 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.333755 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r9f9p\"" Sep 16 13:30:41.333812 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.333797 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 16 13:30:41.335138 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.335122 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d6bps" Sep 16 13:30:41.336274 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.335990 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 16 13:30:41.336274 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.336059 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 16 13:30:41.336274 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.336081 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 16 13:30:41.336274 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.336087 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n47v5\"" Sep 16 13:30:41.336274 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.336011 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 16 13:30:41.336274 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.336067 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 16 13:30:41.336274 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.336008 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 16 13:30:41.337153 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.337138 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.339458 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.339441 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:30:41.339549 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:41.339505 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vg66w" podUID="205445dc-c385-470f-bc60-dfa12d107297" Sep 16 13:30:41.341873 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.341811 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.342175 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342160 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-run-openvswitch\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.342265 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342181 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3cad145-80e4-4662-a1d8-e173c3cddf03-host\") pod \"node-ca-mcc87\" (UID: \"e3cad145-80e4-4662-a1d8-e173c3cddf03\") " pod="openshift-image-registry/node-ca-mcc87" Sep 16 13:30:41.342265 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342198 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-host-kubelet\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.342265 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342213 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-systemd-units\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.342265 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342232 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-host-run-netns\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.342442 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342269 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5g5j\" (UniqueName: \"kubernetes.io/projected/74bcf9e2-c283-4582-bbc2-a321c97c84b9-kube-api-access-m5g5j\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.342442 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342303 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27wzj\" (UniqueName: \"kubernetes.io/projected/e3cad145-80e4-4662-a1d8-e173c3cddf03-kube-api-access-27wzj\") pod \"node-ca-mcc87\" (UID: \"e3cad145-80e4-4662-a1d8-e173c3cddf03\") " pod="openshift-image-registry/node-ca-mcc87" Sep 16 13:30:41.342442 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342327 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2afeee9c-fe7c-4808-a71c-239a0c68a6a5-iptables-alerter-script\") pod \"iptables-alerter-hwrln\" (UID: \"2afeee9c-fe7c-4808-a71c-239a0c68a6a5\") " pod="openshift-network-operator/iptables-alerter-hwrln" Sep 16 13:30:41.342442 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342353 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-log-socket\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.342442 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342404 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2afeee9c-fe7c-4808-a71c-239a0c68a6a5-host-slash\") pod \"iptables-alerter-hwrln\" (UID: \"2afeee9c-fe7c-4808-a71c-239a0c68a6a5\") " pod="openshift-network-operator/iptables-alerter-hwrln" Sep 16 13:30:41.342442 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342429 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.342712 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342465 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/74bcf9e2-c283-4582-bbc2-a321c97c84b9-ovn-node-metrics-cert\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.342712 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342504 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e3cad145-80e4-4662-a1d8-e173c3cddf03-serviceca\") pod \"node-ca-mcc87\" (UID: \"e3cad145-80e4-4662-a1d8-e173c3cddf03\") " pod="openshift-image-registry/node-ca-mcc87" Sep 16 13:30:41.342712 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342535 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-run-systemd\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.342712 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342561 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-node-log\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.342712 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342583 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-host-cni-netd\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.342712 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342606 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/74bcf9e2-c283-4582-bbc2-a321c97c84b9-env-overrides\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.342712 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342627 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-host-slash\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.342712 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342648 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-host-cni-bin\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.342712 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342669 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcw8f\" (UniqueName: \"kubernetes.io/projected/2afeee9c-fe7c-4808-a71c-239a0c68a6a5-kube-api-access-xcw8f\") pod \"iptables-alerter-hwrln\" (UID: \"2afeee9c-fe7c-4808-a71c-239a0c68a6a5\") " pod="openshift-network-operator/iptables-alerter-hwrln" Sep 16 13:30:41.343108 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342713 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-run-ovn\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.343108 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342742 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/74bcf9e2-c283-4582-bbc2-a321c97c84b9-ovnkube-config\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.343108 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342765 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/74bcf9e2-c283-4582-bbc2-a321c97c84b9-ovnkube-script-lib\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.343108 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342785 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-etc-openvswitch\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.343108 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342814 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-var-lib-openvswitch\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.343108 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.342844 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.343923 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.343906 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 16 13:30:41.344453 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.344438 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8zxxf" Sep 16 13:30:41.345248 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.345232 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 16 13:30:41.345341 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.345325 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 16 13:30:41.345571 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.345510 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 16 13:30:41.345571 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.345556 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 16 13:30:41.346503 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.346489 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 16 13:30:41.346809 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.346790 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:30:41.346879 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:41.346856 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s6pv" podUID="2a102e1b-9b9b-45f3-9e7e-33794568bdb9" Sep 16 13:30:41.349454 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.349437 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t9qw9" Sep 16 13:30:41.350229 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.350213 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:30:41.350348 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.350332 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bnr5g\"" Sep 16 13:30:41.350490 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.350475 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 16 13:30:41.350612 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.350598 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 16 13:30:41.350693 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.350678 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-54zlg\"" Sep 16 13:30:41.351135 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.351109 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-prnkq\"" Sep 16 13:30:41.351135 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.351129 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mfwnd\"" Sep 16 13:30:41.351267 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.351153 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 16 13:30:41.352035 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.352020 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 16 13:30:41.353503 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.353098 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" Sep 16 13:30:41.353636 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.353606 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 13:30:41.353636 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.353624 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 16 13:30:41.354013 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.353999 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x2s27\"" Sep 16 13:30:41.355630 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.355528 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:30:41.355630 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:41.355598 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hm6lg" podUID="816a366f-b67b-4679-a323-ab36289dc4b6" Sep 16 13:30:41.355764 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.355734 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 16 13:30:41.358333 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.357728 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 16 13:30:41.358333 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.357761 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 16 13:30:41.358333 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.358135 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-68mtn\"" Sep 16 13:30:41.381526 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.381227 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sftdx" Sep 16 13:30:41.396611 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.396285 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sftdx" Sep 16 13:30:41.411039 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.410991 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-23.ec2.internal" event={"ID":"fc88820cac2d990b85f97c81f96e2d49","Type":"ContainerStarted","Data":"46353344800aea6b4d03c4c3b08e98aca2e0f1c1a9f673f1cc77cc6eb4905609"} Sep 16 13:30:41.412237 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.412210 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal" event={"ID":"b7bf8d3fcfbbc3832a3c68332bac0716","Type":"ContainerStarted","Data":"31d49837f9700c4784d13868dbd7f5963cda2fe0349e8fed4dbc8f03ceacb634"} Sep 16 13:30:41.433518 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.433506 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 16 13:30:41.443966 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.443945 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-host-cni-netd\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.444069 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.443981 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f7d94e4b-754b-40b0-85be-6f1293ec5f07-device-dir\") pod \"aws-ebs-csi-driver-node-p5r5v\" (UID: \"f7d94e4b-754b-40b0-85be-6f1293ec5f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" Sep 16 13:30:41.444069 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444005 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/870338a6-a8f2-46f8-a5e0-b1b17837057a-cnibin\") pod \"multus-additional-cni-plugins-d6bps\" (UID: \"870338a6-a8f2-46f8-a5e0-b1b17837057a\") " pod="openshift-multus/multus-additional-cni-plugins-d6bps" Sep 16 13:30:41.444069 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444028 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/870338a6-a8f2-46f8-a5e0-b1b17837057a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d6bps\" (UID: \"870338a6-a8f2-46f8-a5e0-b1b17837057a\") " pod="openshift-multus/multus-additional-cni-plugins-d6bps" Sep 16 13:30:41.444069 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444050 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-etc-sysctl-conf\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.444069 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444070 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-var-lib-kubelet\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.444297 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444088 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-host-cni-bin\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.444297 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444102 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f7d94e4b-754b-40b0-85be-6f1293ec5f07-socket-dir\") pod \"aws-ebs-csi-driver-node-p5r5v\" (UID: \"f7d94e4b-754b-40b0-85be-6f1293ec5f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" Sep 16 13:30:41.444297 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444116 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-multus-conf-dir\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.444297 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444130 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-etc-sysconfig\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.444297 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444144 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-etc-sysctl-d\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.444297 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444181 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-run-ovn\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.444297 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444206 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/74bcf9e2-c283-4582-bbc2-a321c97c84b9-ovnkube-script-lib\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.444297 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444227 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-host-run-netns\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.444297 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444246 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-etc-openvswitch\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.444297 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444262 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.444297 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444278 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/816a366f-b67b-4679-a323-ab36289dc4b6-metrics-certs\") pod \"network-metrics-daemon-hm6lg\" (UID: \"816a366f-b67b-4679-a323-ab36289dc4b6\") " pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:30:41.444683 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444306 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/870338a6-a8f2-46f8-a5e0-b1b17837057a-system-cni-dir\") pod \"multus-additional-cni-plugins-d6bps\" (UID: \"870338a6-a8f2-46f8-a5e0-b1b17837057a\") " pod="openshift-multus/multus-additional-cni-plugins-d6bps" Sep 16 13:30:41.444683 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444325 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-host-var-lib-kubelet\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.444683 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444338 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-etc-kubernetes\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.444683 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444359 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5695f6e6-c46d-4cd0-b4ab-9ab7c4edc9fc-konnectivity-ca\") pod \"konnectivity-agent-8zxxf\" (UID: \"5695f6e6-c46d-4cd0-b4ab-9ab7c4edc9fc\") " pod="kube-system/konnectivity-agent-8zxxf" Sep 16 13:30:41.444683 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444388 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-run-openvswitch\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.444683 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444407 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3cad145-80e4-4662-a1d8-e173c3cddf03-host\") pod \"node-ca-mcc87\" (UID: \"e3cad145-80e4-4662-a1d8-e173c3cddf03\") " pod="openshift-image-registry/node-ca-mcc87" Sep 16 13:30:41.444683 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444421 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-dbus\") pod \"global-pull-secret-syncer-5s6pv\" (UID: \"2a102e1b-9b9b-45f3-9e7e-33794568bdb9\") " pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:30:41.444683 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444439 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f7d94e4b-754b-40b0-85be-6f1293ec5f07-registration-dir\") pod \"aws-ebs-csi-driver-node-p5r5v\" (UID: \"f7d94e4b-754b-40b0-85be-6f1293ec5f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" Sep 16 13:30:41.444683 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444460 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/870338a6-a8f2-46f8-a5e0-b1b17837057a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d6bps\" (UID: \"870338a6-a8f2-46f8-a5e0-b1b17837057a\") " pod="openshift-multus/multus-additional-cni-plugins-d6bps" Sep 16 13:30:41.444683 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444484 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6khr\" (UniqueName: \"kubernetes.io/projected/870338a6-a8f2-46f8-a5e0-b1b17837057a-kube-api-access-n6khr\") pod \"multus-additional-cni-plugins-d6bps\" (UID: \"870338a6-a8f2-46f8-a5e0-b1b17837057a\") " pod="openshift-multus/multus-additional-cni-plugins-d6bps" Sep 16 13:30:41.444683 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444498 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-host-var-lib-cni-multus\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.444683 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444513 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-lib-modules\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.444683 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444532 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-host-kubelet\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.444683 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444555 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-systemd-units\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.444683 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444579 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5g5j\" (UniqueName: \"kubernetes.io/projected/74bcf9e2-c283-4582-bbc2-a321c97c84b9-kube-api-access-m5g5j\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.444683 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444611 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27wzj\" (UniqueName: \"kubernetes.io/projected/e3cad145-80e4-4662-a1d8-e173c3cddf03-kube-api-access-27wzj\") pod \"node-ca-mcc87\" (UID: \"e3cad145-80e4-4662-a1d8-e173c3cddf03\") " pod="openshift-image-registry/node-ca-mcc87" Sep 16 13:30:41.444683 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444632 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-cnibin\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.445250 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444670 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptcxc\" (UniqueName: \"kubernetes.io/projected/b9f0c124-231d-4677-8696-d7317828eb4f-kube-api-access-ptcxc\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.445250 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444687 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-etc-kubernetes\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.445250 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444706 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-hostroot\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.445250 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444722 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2afeee9c-fe7c-4808-a71c-239a0c68a6a5-host-slash\") pod \"iptables-alerter-hwrln\" (UID: \"2afeee9c-fe7c-4808-a71c-239a0c68a6a5\") " pod="openshift-network-operator/iptables-alerter-hwrln" Sep 16 13:30:41.445250 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444746 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.445250 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444768 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e3cad145-80e4-4662-a1d8-e173c3cddf03-serviceca\") pod \"node-ca-mcc87\" (UID: \"e3cad145-80e4-4662-a1d8-e173c3cddf03\") " pod="openshift-image-registry/node-ca-mcc87" Sep 16 13:30:41.445250 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444787 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f7d94e4b-754b-40b0-85be-6f1293ec5f07-etc-selinux\") pod \"aws-ebs-csi-driver-node-p5r5v\" (UID: \"f7d94e4b-754b-40b0-85be-6f1293ec5f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" Sep 16 13:30:41.445250 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444808 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-etc-systemd\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.445250 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444824 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvgql\" (UniqueName: \"kubernetes.io/projected/cdc69e4f-095b-453f-9114-8199d4a0477c-kube-api-access-lvgql\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.445250 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444837 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-run-systemd\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.445250 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444851 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-node-log\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.445250 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444869 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/74bcf9e2-c283-4582-bbc2-a321c97c84b9-env-overrides\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.445250 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444883 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-run\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.445250 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444896 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-host\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.445250 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444909 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cdc69e4f-095b-453f-9114-8199d4a0477c-etc-tuned\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.445250 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444924 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-host-slash\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.445250 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444938 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-kubelet-config\") pod \"global-pull-secret-syncer-5s6pv\" (UID: \"2a102e1b-9b9b-45f3-9e7e-33794568bdb9\") " pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:30:41.445760 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444953 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f7d94e4b-754b-40b0-85be-6f1293ec5f07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p5r5v\" (UID: \"f7d94e4b-754b-40b0-85be-6f1293ec5f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" Sep 16 13:30:41.445760 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444970 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/870338a6-a8f2-46f8-a5e0-b1b17837057a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d6bps\" (UID: \"870338a6-a8f2-46f8-a5e0-b1b17837057a\") " pod="openshift-multus/multus-additional-cni-plugins-d6bps" Sep 16 13:30:41.445760 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.444988 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-multus-socket-dir-parent\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.445760 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445007 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcw8f\" (UniqueName: \"kubernetes.io/projected/2afeee9c-fe7c-4808-a71c-239a0c68a6a5-kube-api-access-xcw8f\") pod \"iptables-alerter-hwrln\" (UID: \"2afeee9c-fe7c-4808-a71c-239a0c68a6a5\") " pod="openshift-network-operator/iptables-alerter-hwrln" Sep 16 13:30:41.445760 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445021 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/74bcf9e2-c283-4582-bbc2-a321c97c84b9-ovnkube-config\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.445760 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445035 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-multus-cni-dir\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.445760 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445049 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9f0c124-231d-4677-8696-d7317828eb4f-cni-binary-copy\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.445760 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445063 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-host-var-lib-cni-bin\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.445760 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445078 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzvrt\" (UniqueName: \"kubernetes.io/projected/205445dc-c385-470f-bc60-dfa12d107297-kube-api-access-nzvrt\") pod \"network-check-target-vg66w\" (UID: \"205445dc-c385-470f-bc60-dfa12d107297\") " pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:30:41.445760 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445099 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-etc-modprobe-d\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.445760 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445119 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/870338a6-a8f2-46f8-a5e0-b1b17837057a-cni-binary-copy\") pod \"multus-additional-cni-plugins-d6bps\" (UID: \"870338a6-a8f2-46f8-a5e0-b1b17837057a\") " pod="openshift-multus/multus-additional-cni-plugins-d6bps" Sep 16 13:30:41.445760 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445136 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-var-lib-openvswitch\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.445760 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445150 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24582309-5bb1-4d3d-8a18-a1d7803f287a-tmp-dir\") pod \"node-resolver-t9qw9\" (UID: \"24582309-5bb1-4d3d-8a18-a1d7803f287a\") " pod="openshift-dns/node-resolver-t9qw9" Sep 16 13:30:41.445760 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445166 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkr7c\" (UniqueName: \"kubernetes.io/projected/816a366f-b67b-4679-a323-ab36289dc4b6-kube-api-access-lkr7c\") pod \"network-metrics-daemon-hm6lg\" (UID: \"816a366f-b67b-4679-a323-ab36289dc4b6\") " pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:30:41.445760 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445188 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5695f6e6-c46d-4cd0-b4ab-9ab7c4edc9fc-agent-certs\") pod \"konnectivity-agent-8zxxf\" (UID: \"5695f6e6-c46d-4cd0-b4ab-9ab7c4edc9fc\") " pod="kube-system/konnectivity-agent-8zxxf" Sep 16 13:30:41.445760 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445201 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-sys\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.446249 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445215 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-host-run-k8s-cni-cncf-io\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.446249 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445234 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-host-run-netns\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.446249 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445254 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9tm4\" (UniqueName: \"kubernetes.io/projected/24582309-5bb1-4d3d-8a18-a1d7803f287a-kube-api-access-t9tm4\") pod \"node-resolver-t9qw9\" (UID: \"24582309-5bb1-4d3d-8a18-a1d7803f287a\") " pod="openshift-dns/node-resolver-t9qw9" Sep 16 13:30:41.446249 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445290 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4jvk\" (UniqueName: \"kubernetes.io/projected/f7d94e4b-754b-40b0-85be-6f1293ec5f07-kube-api-access-b4jvk\") pod \"aws-ebs-csi-driver-node-p5r5v\" (UID: \"f7d94e4b-754b-40b0-85be-6f1293ec5f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" Sep 16 13:30:41.446249 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445313 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b9f0c124-231d-4677-8696-d7317828eb4f-multus-daemon-config\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.446249 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445329 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2afeee9c-fe7c-4808-a71c-239a0c68a6a5-iptables-alerter-script\") pod \"iptables-alerter-hwrln\" (UID: \"2afeee9c-fe7c-4808-a71c-239a0c68a6a5\") " pod="openshift-network-operator/iptables-alerter-hwrln" Sep 16 13:30:41.446249 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445345 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-log-socket\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.446249 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445361 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-system-cni-dir\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.446249 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445392 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-host-run-multus-certs\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.446249 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445414 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cdc69e4f-095b-453f-9114-8199d4a0477c-tmp\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.446249 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445435 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/74bcf9e2-c283-4582-bbc2-a321c97c84b9-ovn-node-metrics-cert\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.446249 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445450 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-original-pull-secret\") pod \"global-pull-secret-syncer-5s6pv\" (UID: \"2a102e1b-9b9b-45f3-9e7e-33794568bdb9\") " pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:30:41.446249 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445465 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24582309-5bb1-4d3d-8a18-a1d7803f287a-hosts-file\") pod \"node-resolver-t9qw9\" (UID: \"24582309-5bb1-4d3d-8a18-a1d7803f287a\") " pod="openshift-dns/node-resolver-t9qw9" Sep 16 13:30:41.446249 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445480 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f7d94e4b-754b-40b0-85be-6f1293ec5f07-sys-fs\") pod \"aws-ebs-csi-driver-node-p5r5v\" (UID: \"f7d94e4b-754b-40b0-85be-6f1293ec5f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" Sep 16 13:30:41.446249 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445535 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-host-cni-netd\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.446249 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445589 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-host-cni-bin\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.446249 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.445638 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-run-ovn\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.446805 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.446041 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/74bcf9e2-c283-4582-bbc2-a321c97c84b9-ovnkube-script-lib\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.446805 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.446129 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-etc-openvswitch\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.446805 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.446161 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-host-run-ovn-kubernetes\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.446805 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.446167 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-host-slash\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.446805 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.446725 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/74bcf9e2-c283-4582-bbc2-a321c97c84b9-env-overrides\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.446805 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.446785 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/74bcf9e2-c283-4582-bbc2-a321c97c84b9-ovnkube-config\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.446984 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.446834 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-var-lib-openvswitch\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.446984 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.446865 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-run-openvswitch\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.446984 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.446887 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3cad145-80e4-4662-a1d8-e173c3cddf03-host\") pod \"node-ca-mcc87\" (UID: \"e3cad145-80e4-4662-a1d8-e173c3cddf03\") " pod="openshift-image-registry/node-ca-mcc87" Sep 16 13:30:41.446984 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.446906 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-host-run-netns\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.448169 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.447155 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-systemd-units\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.448169 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.447202 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-host-kubelet\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.448169 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.447232 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-log-socket\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.448169 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.447237 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2afeee9c-fe7c-4808-a71c-239a0c68a6a5-iptables-alerter-script\") pod \"iptables-alerter-hwrln\" (UID: \"2afeee9c-fe7c-4808-a71c-239a0c68a6a5\") " pod="openshift-network-operator/iptables-alerter-hwrln" Sep 16 13:30:41.448169 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.447241 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.448169 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.447294 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-run-systemd\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.448169 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.447365 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2afeee9c-fe7c-4808-a71c-239a0c68a6a5-host-slash\") pod \"iptables-alerter-hwrln\" (UID: \"2afeee9c-fe7c-4808-a71c-239a0c68a6a5\") " pod="openshift-network-operator/iptables-alerter-hwrln" Sep 16 13:30:41.448169 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.447407 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/74bcf9e2-c283-4582-bbc2-a321c97c84b9-node-log\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.448169 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.447622 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e3cad145-80e4-4662-a1d8-e173c3cddf03-serviceca\") pod \"node-ca-mcc87\" (UID: \"e3cad145-80e4-4662-a1d8-e173c3cddf03\") " pod="openshift-image-registry/node-ca-mcc87" Sep 16 13:30:41.448169 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.447911 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 16 13:30:41.452719 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.452675 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/74bcf9e2-c283-4582-bbc2-a321c97c84b9-ovn-node-metrics-cert\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.459055 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.459037 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27wzj\" (UniqueName: \"kubernetes.io/projected/e3cad145-80e4-4662-a1d8-e173c3cddf03-kube-api-access-27wzj\") pod \"node-ca-mcc87\" (UID: \"e3cad145-80e4-4662-a1d8-e173c3cddf03\") " pod="openshift-image-registry/node-ca-mcc87" Sep 16 13:30:41.460756 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.460723 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcw8f\" (UniqueName: \"kubernetes.io/projected/2afeee9c-fe7c-4808-a71c-239a0c68a6a5-kube-api-access-xcw8f\") pod \"iptables-alerter-hwrln\" (UID: \"2afeee9c-fe7c-4808-a71c-239a0c68a6a5\") " pod="openshift-network-operator/iptables-alerter-hwrln" Sep 16 13:30:41.461886 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.461866 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5g5j\" (UniqueName: \"kubernetes.io/projected/74bcf9e2-c283-4582-bbc2-a321c97c84b9-kube-api-access-m5g5j\") pod \"ovnkube-node-2k2tg\" (UID: \"74bcf9e2-c283-4582-bbc2-a321c97c84b9\") " pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.546542 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.546510 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f7d94e4b-754b-40b0-85be-6f1293ec5f07-device-dir\") pod \"aws-ebs-csi-driver-node-p5r5v\" (UID: \"f7d94e4b-754b-40b0-85be-6f1293ec5f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" Sep 16 13:30:41.546697 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.546552 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/870338a6-a8f2-46f8-a5e0-b1b17837057a-cnibin\") pod \"multus-additional-cni-plugins-d6bps\" (UID: \"870338a6-a8f2-46f8-a5e0-b1b17837057a\") " pod="openshift-multus/multus-additional-cni-plugins-d6bps" Sep 16 13:30:41.546697 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.546578 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/870338a6-a8f2-46f8-a5e0-b1b17837057a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d6bps\" (UID: \"870338a6-a8f2-46f8-a5e0-b1b17837057a\") " pod="openshift-multus/multus-additional-cni-plugins-d6bps" Sep 16 13:30:41.546697 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.546609 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-etc-sysctl-conf\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.546697 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.546634 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-var-lib-kubelet\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.546697 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.546637 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f7d94e4b-754b-40b0-85be-6f1293ec5f07-device-dir\") pod \"aws-ebs-csi-driver-node-p5r5v\" (UID: \"f7d94e4b-754b-40b0-85be-6f1293ec5f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" Sep 16 13:30:41.546697 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.546658 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f7d94e4b-754b-40b0-85be-6f1293ec5f07-socket-dir\") pod \"aws-ebs-csi-driver-node-p5r5v\" (UID: \"f7d94e4b-754b-40b0-85be-6f1293ec5f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" Sep 16 13:30:41.546697 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.546681 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-multus-conf-dir\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.547030 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.546702 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-etc-sysconfig\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.547030 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.546724 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-etc-sysctl-d\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.547030 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.546748 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-host-run-netns\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.547030 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.546768 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-etc-sysctl-conf\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.547030 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.546773 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/816a366f-b67b-4679-a323-ab36289dc4b6-metrics-certs\") pod \"network-metrics-daemon-hm6lg\" (UID: \"816a366f-b67b-4679-a323-ab36289dc4b6\") " pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:30:41.547030 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.546832 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/870338a6-a8f2-46f8-a5e0-b1b17837057a-system-cni-dir\") pod \"multus-additional-cni-plugins-d6bps\" (UID: \"870338a6-a8f2-46f8-a5e0-b1b17837057a\") " pod="openshift-multus/multus-additional-cni-plugins-d6bps" Sep 16 13:30:41.547030 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:41.546851 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:30:41.547030 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.546857 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-host-var-lib-kubelet\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.547030 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.546881 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-etc-kubernetes\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.547030 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:41.546921 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/816a366f-b67b-4679-a323-ab36289dc4b6-metrics-certs podName:816a366f-b67b-4679-a323-ab36289dc4b6 nodeName:}" failed. No retries permitted until 2026-09-16 13:30:42.046895927 +0000 UTC m=+3.091198902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/816a366f-b67b-4679-a323-ab36289dc4b6-metrics-certs") pod "network-metrics-daemon-hm6lg" (UID: "816a366f-b67b-4679-a323-ab36289dc4b6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:30:41.547030 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.546935 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5695f6e6-c46d-4cd0-b4ab-9ab7c4edc9fc-konnectivity-ca\") pod \"konnectivity-agent-8zxxf\" (UID: \"5695f6e6-c46d-4cd0-b4ab-9ab7c4edc9fc\") " pod="kube-system/konnectivity-agent-8zxxf" Sep 16 13:30:41.547030 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.546944 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-etc-kubernetes\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.547030 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.546954 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-dbus\") pod \"global-pull-secret-syncer-5s6pv\" (UID: \"2a102e1b-9b9b-45f3-9e7e-33794568bdb9\") " pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:30:41.547030 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.546978 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f7d94e4b-754b-40b0-85be-6f1293ec5f07-registration-dir\") pod \"aws-ebs-csi-driver-node-p5r5v\" (UID: \"f7d94e4b-754b-40b0-85be-6f1293ec5f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" Sep 16 13:30:41.547030 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.546996 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-dbus\") pod \"global-pull-secret-syncer-5s6pv\" (UID: \"2a102e1b-9b9b-45f3-9e7e-33794568bdb9\") " pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:30:41.547030 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547003 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/870338a6-a8f2-46f8-a5e0-b1b17837057a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d6bps\" (UID: \"870338a6-a8f2-46f8-a5e0-b1b17837057a\") " pod="openshift-multus/multus-additional-cni-plugins-d6bps" Sep 16 13:30:41.547710 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547034 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-var-lib-kubelet\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.547710 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547050 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6khr\" (UniqueName: \"kubernetes.io/projected/870338a6-a8f2-46f8-a5e0-b1b17837057a-kube-api-access-n6khr\") pod \"multus-additional-cni-plugins-d6bps\" (UID: \"870338a6-a8f2-46f8-a5e0-b1b17837057a\") " pod="openshift-multus/multus-additional-cni-plugins-d6bps" Sep 16 13:30:41.547710 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547074 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-host-var-lib-cni-multus\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.547710 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547096 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-lib-modules\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.547710 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547123 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f7d94e4b-754b-40b0-85be-6f1293ec5f07-socket-dir\") pod \"aws-ebs-csi-driver-node-p5r5v\" (UID: \"f7d94e4b-754b-40b0-85be-6f1293ec5f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" Sep 16 13:30:41.547710 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547125 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-cnibin\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.547710 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547151 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptcxc\" (UniqueName: \"kubernetes.io/projected/b9f0c124-231d-4677-8696-d7317828eb4f-kube-api-access-ptcxc\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.547710 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547163 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-cnibin\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.547710 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547165 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-etc-kubernetes\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.547710 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547193 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/870338a6-a8f2-46f8-a5e0-b1b17837057a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d6bps\" (UID: \"870338a6-a8f2-46f8-a5e0-b1b17837057a\") " pod="openshift-multus/multus-additional-cni-plugins-d6bps" Sep 16 13:30:41.547710 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547227 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-etc-kubernetes\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.547710 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547233 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-hostroot\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.547710 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547260 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-etc-sysconfig\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.547710 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547264 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f7d94e4b-754b-40b0-85be-6f1293ec5f07-etc-selinux\") pod \"aws-ebs-csi-driver-node-p5r5v\" (UID: \"f7d94e4b-754b-40b0-85be-6f1293ec5f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" Sep 16 13:30:41.547710 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.546643 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/870338a6-a8f2-46f8-a5e0-b1b17837057a-cnibin\") pod \"multus-additional-cni-plugins-d6bps\" (UID: \"870338a6-a8f2-46f8-a5e0-b1b17837057a\") " pod="openshift-multus/multus-additional-cni-plugins-d6bps" Sep 16 13:30:41.547710 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547319 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-host-run-netns\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.547710 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547339 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-multus-conf-dir\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.547710 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547404 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f7d94e4b-754b-40b0-85be-6f1293ec5f07-etc-selinux\") pod \"aws-ebs-csi-driver-node-p5r5v\" (UID: \"f7d94e4b-754b-40b0-85be-6f1293ec5f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" Sep 16 13:30:41.548502 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547593 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-host-var-lib-cni-multus\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.548502 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547461 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-etc-systemd\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.548502 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547527 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-etc-systemd\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.548502 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547557 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/870338a6-a8f2-46f8-a5e0-b1b17837057a-system-cni-dir\") pod \"multus-additional-cni-plugins-d6bps\" (UID: \"870338a6-a8f2-46f8-a5e0-b1b17837057a\") " pod="openshift-multus/multus-additional-cni-plugins-d6bps" Sep 16 13:30:41.548502 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547624 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-hostroot\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.548502 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547439 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-host-var-lib-kubelet\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.548502 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547557 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-etc-sysctl-d\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.548502 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547596 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f7d94e4b-754b-40b0-85be-6f1293ec5f07-registration-dir\") pod \"aws-ebs-csi-driver-node-p5r5v\" (UID: \"f7d94e4b-754b-40b0-85be-6f1293ec5f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" Sep 16 13:30:41.548502 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547698 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvgql\" (UniqueName: \"kubernetes.io/projected/cdc69e4f-095b-453f-9114-8199d4a0477c-kube-api-access-lvgql\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.548502 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547739 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-lib-modules\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.548502 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547749 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-run\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.548502 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547773 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-host\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.548502 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547785 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5695f6e6-c46d-4cd0-b4ab-9ab7c4edc9fc-konnectivity-ca\") pod \"konnectivity-agent-8zxxf\" (UID: \"5695f6e6-c46d-4cd0-b4ab-9ab7c4edc9fc\") " pod="kube-system/konnectivity-agent-8zxxf" Sep 16 13:30:41.548502 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547818 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cdc69e4f-095b-453f-9114-8199d4a0477c-etc-tuned\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.548502 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547853 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-kubelet-config\") pod \"global-pull-secret-syncer-5s6pv\" (UID: \"2a102e1b-9b9b-45f3-9e7e-33794568bdb9\") " pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:30:41.548502 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547872 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-host\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.548502 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547876 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-run\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.548502 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547889 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f7d94e4b-754b-40b0-85be-6f1293ec5f07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p5r5v\" (UID: \"f7d94e4b-754b-40b0-85be-6f1293ec5f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" Sep 16 13:30:41.549129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547922 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/870338a6-a8f2-46f8-a5e0-b1b17837057a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d6bps\" (UID: \"870338a6-a8f2-46f8-a5e0-b1b17837057a\") " pod="openshift-multus/multus-additional-cni-plugins-d6bps" Sep 16 13:30:41.549129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547922 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-kubelet-config\") pod \"global-pull-secret-syncer-5s6pv\" (UID: \"2a102e1b-9b9b-45f3-9e7e-33794568bdb9\") " pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:30:41.549129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547938 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f7d94e4b-754b-40b0-85be-6f1293ec5f07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p5r5v\" (UID: \"f7d94e4b-754b-40b0-85be-6f1293ec5f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" Sep 16 13:30:41.549129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.547963 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/870338a6-a8f2-46f8-a5e0-b1b17837057a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d6bps\" (UID: \"870338a6-a8f2-46f8-a5e0-b1b17837057a\") " pod="openshift-multus/multus-additional-cni-plugins-d6bps" Sep 16 13:30:41.549129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548010 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-multus-socket-dir-parent\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.549129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548046 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-multus-cni-dir\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.549129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548088 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9f0c124-231d-4677-8696-d7317828eb4f-cni-binary-copy\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.549129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548110 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-host-var-lib-cni-bin\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.549129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548121 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-multus-socket-dir-parent\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.549129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548132 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzvrt\" (UniqueName: \"kubernetes.io/projected/205445dc-c385-470f-bc60-dfa12d107297-kube-api-access-nzvrt\") pod \"network-check-target-vg66w\" (UID: \"205445dc-c385-470f-bc60-dfa12d107297\") " pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:30:41.549129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548155 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-etc-modprobe-d\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.549129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548167 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-multus-cni-dir\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.549129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548176 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/870338a6-a8f2-46f8-a5e0-b1b17837057a-cni-binary-copy\") pod \"multus-additional-cni-plugins-d6bps\" (UID: \"870338a6-a8f2-46f8-a5e0-b1b17837057a\") " pod="openshift-multus/multus-additional-cni-plugins-d6bps" Sep 16 13:30:41.549129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548223 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-host-var-lib-cni-bin\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.549129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548249 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24582309-5bb1-4d3d-8a18-a1d7803f287a-tmp-dir\") pod \"node-resolver-t9qw9\" (UID: \"24582309-5bb1-4d3d-8a18-a1d7803f287a\") " pod="openshift-dns/node-resolver-t9qw9" Sep 16 13:30:41.549129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548126 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/870338a6-a8f2-46f8-a5e0-b1b17837057a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d6bps\" (UID: \"870338a6-a8f2-46f8-a5e0-b1b17837057a\") " pod="openshift-multus/multus-additional-cni-plugins-d6bps" Sep 16 13:30:41.549129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548427 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-etc-modprobe-d\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.549693 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548446 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkr7c\" (UniqueName: \"kubernetes.io/projected/816a366f-b67b-4679-a323-ab36289dc4b6-kube-api-access-lkr7c\") pod \"network-metrics-daemon-hm6lg\" (UID: \"816a366f-b67b-4679-a323-ab36289dc4b6\") " pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:30:41.549693 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548488 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5695f6e6-c46d-4cd0-b4ab-9ab7c4edc9fc-agent-certs\") pod \"konnectivity-agent-8zxxf\" (UID: \"5695f6e6-c46d-4cd0-b4ab-9ab7c4edc9fc\") " pod="kube-system/konnectivity-agent-8zxxf" Sep 16 13:30:41.549693 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548512 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-sys\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.549693 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548690 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9f0c124-231d-4677-8696-d7317828eb4f-cni-binary-copy\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.549693 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548708 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24582309-5bb1-4d3d-8a18-a1d7803f287a-tmp-dir\") pod \"node-resolver-t9qw9\" (UID: \"24582309-5bb1-4d3d-8a18-a1d7803f287a\") " pod="openshift-dns/node-resolver-t9qw9" Sep 16 13:30:41.549693 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548697 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/870338a6-a8f2-46f8-a5e0-b1b17837057a-cni-binary-copy\") pod \"multus-additional-cni-plugins-d6bps\" (UID: \"870338a6-a8f2-46f8-a5e0-b1b17837057a\") " pod="openshift-multus/multus-additional-cni-plugins-d6bps" Sep 16 13:30:41.549693 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548755 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cdc69e4f-095b-453f-9114-8199d4a0477c-sys\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.549693 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548759 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-host-run-k8s-cni-cncf-io\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.549693 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548867 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9tm4\" (UniqueName: \"kubernetes.io/projected/24582309-5bb1-4d3d-8a18-a1d7803f287a-kube-api-access-t9tm4\") pod \"node-resolver-t9qw9\" (UID: \"24582309-5bb1-4d3d-8a18-a1d7803f287a\") " pod="openshift-dns/node-resolver-t9qw9" Sep 16 13:30:41.549693 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548901 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4jvk\" (UniqueName: \"kubernetes.io/projected/f7d94e4b-754b-40b0-85be-6f1293ec5f07-kube-api-access-b4jvk\") pod \"aws-ebs-csi-driver-node-p5r5v\" (UID: \"f7d94e4b-754b-40b0-85be-6f1293ec5f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" Sep 16 13:30:41.549693 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548924 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b9f0c124-231d-4677-8696-d7317828eb4f-multus-daemon-config\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.549693 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548969 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-system-cni-dir\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.549693 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.548993 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-host-run-multus-certs\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.549693 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.549090 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-system-cni-dir\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.549693 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.549263 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-host-run-k8s-cni-cncf-io\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.549693 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.549370 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b9f0c124-231d-4677-8696-d7317828eb4f-multus-daemon-config\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.549693 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.549420 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cdc69e4f-095b-453f-9114-8199d4a0477c-tmp\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.549693 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.549449 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b9f0c124-231d-4677-8696-d7317828eb4f-host-run-multus-certs\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.550662 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.549494 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-original-pull-secret\") pod \"global-pull-secret-syncer-5s6pv\" (UID: \"2a102e1b-9b9b-45f3-9e7e-33794568bdb9\") " pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:30:41.550662 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.549551 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24582309-5bb1-4d3d-8a18-a1d7803f287a-hosts-file\") pod \"node-resolver-t9qw9\" (UID: \"24582309-5bb1-4d3d-8a18-a1d7803f287a\") " pod="openshift-dns/node-resolver-t9qw9" Sep 16 13:30:41.550662 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:41.549635 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:30:41.550662 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.549641 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f7d94e4b-754b-40b0-85be-6f1293ec5f07-sys-fs\") pod \"aws-ebs-csi-driver-node-p5r5v\" (UID: \"f7d94e4b-754b-40b0-85be-6f1293ec5f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" Sep 16 13:30:41.550662 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.549687 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24582309-5bb1-4d3d-8a18-a1d7803f287a-hosts-file\") pod \"node-resolver-t9qw9\" (UID: \"24582309-5bb1-4d3d-8a18-a1d7803f287a\") " pod="openshift-dns/node-resolver-t9qw9" Sep 16 13:30:41.550662 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.549756 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f7d94e4b-754b-40b0-85be-6f1293ec5f07-sys-fs\") pod \"aws-ebs-csi-driver-node-p5r5v\" (UID: \"f7d94e4b-754b-40b0-85be-6f1293ec5f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" Sep 16 13:30:41.550662 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:41.550039 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-original-pull-secret podName:2a102e1b-9b9b-45f3-9e7e-33794568bdb9 nodeName:}" failed. No retries permitted until 2026-09-16 13:30:42.050016379 +0000 UTC m=+3.094319326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-original-pull-secret") pod "global-pull-secret-syncer-5s6pv" (UID: "2a102e1b-9b9b-45f3-9e7e-33794568bdb9") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:30:41.550662 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.550222 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cdc69e4f-095b-453f-9114-8199d4a0477c-etc-tuned\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.551429 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.551407 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5695f6e6-c46d-4cd0-b4ab-9ab7c4edc9fc-agent-certs\") pod \"konnectivity-agent-8zxxf\" (UID: \"5695f6e6-c46d-4cd0-b4ab-9ab7c4edc9fc\") " pod="kube-system/konnectivity-agent-8zxxf" Sep 16 13:30:41.551803 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.551774 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cdc69e4f-095b-453f-9114-8199d4a0477c-tmp\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.561305 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:41.561284 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:30:41.561305 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:41.561304 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:30:41.561463 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:41.561313 2562 projected.go:196] Error preparing data for projected volume kube-api-access-nzvrt for pod openshift-network-diagnostics/network-check-target-vg66w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:30:41.561463 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:41.561403 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/205445dc-c385-470f-bc60-dfa12d107297-kube-api-access-nzvrt podName:205445dc-c385-470f-bc60-dfa12d107297 nodeName:}" failed. No retries permitted until 2026-09-16 13:30:42.061370242 +0000 UTC m=+3.105673214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nzvrt" (UniqueName: "kubernetes.io/projected/205445dc-c385-470f-bc60-dfa12d107297-kube-api-access-nzvrt") pod "network-check-target-vg66w" (UID: "205445dc-c385-470f-bc60-dfa12d107297") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:30:41.561463 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.561428 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptcxc\" (UniqueName: \"kubernetes.io/projected/b9f0c124-231d-4677-8696-d7317828eb4f-kube-api-access-ptcxc\") pod \"multus-dwt2h\" (UID: \"b9f0c124-231d-4677-8696-d7317828eb4f\") " pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.566108 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.566056 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9tm4\" (UniqueName: \"kubernetes.io/projected/24582309-5bb1-4d3d-8a18-a1d7803f287a-kube-api-access-t9tm4\") pod \"node-resolver-t9qw9\" (UID: \"24582309-5bb1-4d3d-8a18-a1d7803f287a\") " pod="openshift-dns/node-resolver-t9qw9" Sep 16 13:30:41.568799 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.568770 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4jvk\" (UniqueName: \"kubernetes.io/projected/f7d94e4b-754b-40b0-85be-6f1293ec5f07-kube-api-access-b4jvk\") pod \"aws-ebs-csi-driver-node-p5r5v\" (UID: \"f7d94e4b-754b-40b0-85be-6f1293ec5f07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" Sep 16 13:30:41.569314 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.569274 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6khr\" (UniqueName: \"kubernetes.io/projected/870338a6-a8f2-46f8-a5e0-b1b17837057a-kube-api-access-n6khr\") pod \"multus-additional-cni-plugins-d6bps\" (UID: \"870338a6-a8f2-46f8-a5e0-b1b17837057a\") " pod="openshift-multus/multus-additional-cni-plugins-d6bps" Sep 16 13:30:41.569837 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.569817 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvgql\" (UniqueName: \"kubernetes.io/projected/cdc69e4f-095b-453f-9114-8199d4a0477c-kube-api-access-lvgql\") pod \"tuned-kmpb5\" (UID: \"cdc69e4f-095b-453f-9114-8199d4a0477c\") " pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.571858 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.571838 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkr7c\" (UniqueName: \"kubernetes.io/projected/816a366f-b67b-4679-a323-ab36289dc4b6-kube-api-access-lkr7c\") pod \"network-metrics-daemon-hm6lg\" (UID: \"816a366f-b67b-4679-a323-ab36289dc4b6\") " pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:30:41.638689 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.638666 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hwrln" Sep 16 13:30:41.645618 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.645599 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:30:41.648342 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:41.648320 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2afeee9c_fe7c_4808_a71c_239a0c68a6a5.slice/crio-8c020565ace5daf93805c78333a7e779b4b1b254c42a4aa70d7886509e2d6a08 WatchSource:0}: Error finding container 8c020565ace5daf93805c78333a7e779b4b1b254c42a4aa70d7886509e2d6a08: Status 404 returned error can't find the container with id 8c020565ace5daf93805c78333a7e779b4b1b254c42a4aa70d7886509e2d6a08 Sep 16 13:30:41.652206 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.652190 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:30:41.654623 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.654606 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mcc87" Sep 16 13:30:41.654920 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:41.654898 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74bcf9e2_c283_4582_bbc2_a321c97c84b9.slice/crio-54e6f8f30f086e264944c5d9b348c23b098f9c64815cf82646279c4d90ab2735 WatchSource:0}: Error finding container 54e6f8f30f086e264944c5d9b348c23b098f9c64815cf82646279c4d90ab2735: Status 404 returned error can't find the container with id 54e6f8f30f086e264944c5d9b348c23b098f9c64815cf82646279c4d90ab2735 Sep 16 13:30:41.663623 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.663596 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d6bps" Sep 16 13:30:41.664168 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:41.664150 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3cad145_80e4_4662_a1d8_e173c3cddf03.slice/crio-57144b23c39912d1d497b5d1e8b5393b806048fc6acad7c66ffd7d39d1240f2a WatchSource:0}: Error finding container 57144b23c39912d1d497b5d1e8b5393b806048fc6acad7c66ffd7d39d1240f2a: Status 404 returned error can't find the container with id 57144b23c39912d1d497b5d1e8b5393b806048fc6acad7c66ffd7d39d1240f2a Sep 16 13:30:41.669610 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.669590 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dwt2h" Sep 16 13:30:41.673089 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:41.673072 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870338a6_a8f2_46f8_a5e0_b1b17837057a.slice/crio-3f4ce9c587f33e7f925fe63f5b3a0ce032c8f6136f093e224a7b1615afdce021 WatchSource:0}: Error finding container 3f4ce9c587f33e7f925fe63f5b3a0ce032c8f6136f093e224a7b1615afdce021: Status 404 returned error can't find the container with id 3f4ce9c587f33e7f925fe63f5b3a0ce032c8f6136f093e224a7b1615afdce021 Sep 16 13:30:41.674655 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.674636 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" Sep 16 13:30:41.678216 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:41.678195 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f0c124_231d_4677_8696_d7317828eb4f.slice/crio-0e6bf96b2824e5ddfd3d5a60659cf4bae89abfb96d5eae3df60805ec36a4c857 WatchSource:0}: Error finding container 0e6bf96b2824e5ddfd3d5a60659cf4bae89abfb96d5eae3df60805ec36a4c857: Status 404 returned error can't find the container with id 0e6bf96b2824e5ddfd3d5a60659cf4bae89abfb96d5eae3df60805ec36a4c857 Sep 16 13:30:41.682525 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.682508 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8zxxf" Sep 16 13:30:41.682942 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:41.682924 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc69e4f_095b_453f_9114_8199d4a0477c.slice/crio-db0f1a013033c9fcf0754176e218ff31a767691abe932e67ccab94b749cba637 WatchSource:0}: Error finding container db0f1a013033c9fcf0754176e218ff31a767691abe932e67ccab94b749cba637: Status 404 returned error can't find the container with id db0f1a013033c9fcf0754176e218ff31a767691abe932e67ccab94b749cba637 Sep 16 13:30:41.689454 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.689433 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t9qw9" Sep 16 13:30:41.691848 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:41.691828 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5695f6e6_c46d_4cd0_b4ab_9ab7c4edc9fc.slice/crio-458c2b5a10990b1954d80a6e42a3ee92c5cb721391ebd5da3befd204e31a504c WatchSource:0}: Error finding container 458c2b5a10990b1954d80a6e42a3ee92c5cb721391ebd5da3befd204e31a504c: Status 404 returned error can't find the container with id 458c2b5a10990b1954d80a6e42a3ee92c5cb721391ebd5da3befd204e31a504c Sep 16 13:30:41.695068 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:41.694980 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" Sep 16 13:30:41.698638 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:41.698615 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24582309_5bb1_4d3d_8a18_a1d7803f287a.slice/crio-978e7af8746d06138b27e3a8d62549705d253705eacd3e078d518ee0c8c4a97d WatchSource:0}: Error finding container 978e7af8746d06138b27e3a8d62549705d253705eacd3e078d518ee0c8c4a97d: Status 404 returned error can't find the container with id 978e7af8746d06138b27e3a8d62549705d253705eacd3e078d518ee0c8c4a97d Sep 16 13:30:41.705016 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:30:41.704992 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d94e4b_754b_40b0_85be_6f1293ec5f07.slice/crio-4ec2e4f7f4a9284a7d45a96c0c43b1090067eb893f518119a67fe4b498fa5c51 WatchSource:0}: Error finding container 4ec2e4f7f4a9284a7d45a96c0c43b1090067eb893f518119a67fe4b498fa5c51: Status 404 returned error can't find the container with id 4ec2e4f7f4a9284a7d45a96c0c43b1090067eb893f518119a67fe4b498fa5c51 Sep 16 13:30:42.052837 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:42.052761 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-original-pull-secret\") pod \"global-pull-secret-syncer-5s6pv\" (UID: \"2a102e1b-9b9b-45f3-9e7e-33794568bdb9\") " pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:30:42.052837 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:42.052802 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/816a366f-b67b-4679-a323-ab36289dc4b6-metrics-certs\") pod \"network-metrics-daemon-hm6lg\" (UID: \"816a366f-b67b-4679-a323-ab36289dc4b6\") " pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:30:42.053014 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:42.052926 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:30:42.053014 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:42.052926 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:30:42.053014 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:42.052990 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/816a366f-b67b-4679-a323-ab36289dc4b6-metrics-certs podName:816a366f-b67b-4679-a323-ab36289dc4b6 nodeName:}" failed. No retries permitted until 2026-09-16 13:30:43.052971597 +0000 UTC m=+4.097274546 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/816a366f-b67b-4679-a323-ab36289dc4b6-metrics-certs") pod "network-metrics-daemon-hm6lg" (UID: "816a366f-b67b-4679-a323-ab36289dc4b6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:30:42.053014 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:42.053009 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-original-pull-secret podName:2a102e1b-9b9b-45f3-9e7e-33794568bdb9 nodeName:}" failed. No retries permitted until 2026-09-16 13:30:43.052998726 +0000 UTC m=+4.097301678 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-original-pull-secret") pod "global-pull-secret-syncer-5s6pv" (UID: "2a102e1b-9b9b-45f3-9e7e-33794568bdb9") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:30:42.153731 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:42.153698 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzvrt\" (UniqueName: \"kubernetes.io/projected/205445dc-c385-470f-bc60-dfa12d107297-kube-api-access-nzvrt\") pod \"network-check-target-vg66w\" (UID: \"205445dc-c385-470f-bc60-dfa12d107297\") " pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:30:42.153904 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:42.153886 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:30:42.153968 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:42.153914 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:30:42.153968 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:42.153928 2562 projected.go:196] Error preparing data for projected volume kube-api-access-nzvrt for pod openshift-network-diagnostics/network-check-target-vg66w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:30:42.154059 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:42.153992 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/205445dc-c385-470f-bc60-dfa12d107297-kube-api-access-nzvrt podName:205445dc-c385-470f-bc60-dfa12d107297 nodeName:}" failed. No retries permitted until 2026-09-16 13:30:43.153974101 +0000 UTC m=+4.198277047 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzvrt" (UniqueName: "kubernetes.io/projected/205445dc-c385-470f-bc60-dfa12d107297-kube-api-access-nzvrt") pod "network-check-target-vg66w" (UID: "205445dc-c385-470f-bc60-dfa12d107297") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:30:42.399451 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:42.399362 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 13:25:41 +0000 UTC" deadline="2028-06-17 13:30:18.907056612 +0000 UTC" Sep 16 13:30:42.399451 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:42.399422 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15359h59m36.50765667s" Sep 16 13:30:42.407790 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:42.407765 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:30:42.407925 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:42.407876 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vg66w" podUID="205445dc-c385-470f-bc60-dfa12d107297" Sep 16 13:30:42.408215 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:42.408197 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:30:42.408310 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:42.408290 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s6pv" podUID="2a102e1b-9b9b-45f3-9e7e-33794568bdb9" Sep 16 13:30:42.415178 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:42.415146 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t9qw9" event={"ID":"24582309-5bb1-4d3d-8a18-a1d7803f287a","Type":"ContainerStarted","Data":"978e7af8746d06138b27e3a8d62549705d253705eacd3e078d518ee0c8c4a97d"} Sep 16 13:30:42.416436 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:42.416412 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mcc87" event={"ID":"e3cad145-80e4-4662-a1d8-e173c3cddf03","Type":"ContainerStarted","Data":"57144b23c39912d1d497b5d1e8b5393b806048fc6acad7c66ffd7d39d1240f2a"} Sep 16 13:30:42.417702 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:42.417682 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" event={"ID":"74bcf9e2-c283-4582-bbc2-a321c97c84b9","Type":"ContainerStarted","Data":"54e6f8f30f086e264944c5d9b348c23b098f9c64815cf82646279c4d90ab2735"} Sep 16 13:30:42.418779 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:42.418758 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" event={"ID":"f7d94e4b-754b-40b0-85be-6f1293ec5f07","Type":"ContainerStarted","Data":"4ec2e4f7f4a9284a7d45a96c0c43b1090067eb893f518119a67fe4b498fa5c51"} Sep 16 13:30:42.420010 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:42.419990 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8zxxf" event={"ID":"5695f6e6-c46d-4cd0-b4ab-9ab7c4edc9fc","Type":"ContainerStarted","Data":"458c2b5a10990b1954d80a6e42a3ee92c5cb721391ebd5da3befd204e31a504c"} Sep 16 13:30:42.421223 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:42.421203 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" event={"ID":"cdc69e4f-095b-453f-9114-8199d4a0477c","Type":"ContainerStarted","Data":"db0f1a013033c9fcf0754176e218ff31a767691abe932e67ccab94b749cba637"} Sep 16 13:30:42.422427 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:42.422372 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dwt2h" event={"ID":"b9f0c124-231d-4677-8696-d7317828eb4f","Type":"ContainerStarted","Data":"0e6bf96b2824e5ddfd3d5a60659cf4bae89abfb96d5eae3df60805ec36a4c857"} Sep 16 13:30:42.423612 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:42.423591 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6bps" event={"ID":"870338a6-a8f2-46f8-a5e0-b1b17837057a","Type":"ContainerStarted","Data":"3f4ce9c587f33e7f925fe63f5b3a0ce032c8f6136f093e224a7b1615afdce021"} Sep 16 13:30:42.424812 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:42.424791 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hwrln" event={"ID":"2afeee9c-fe7c-4808-a71c-239a0c68a6a5","Type":"ContainerStarted","Data":"8c020565ace5daf93805c78333a7e779b4b1b254c42a4aa70d7886509e2d6a08"} Sep 16 13:30:42.789431 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:42.789341 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 13:30:43.059739 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:43.059714 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/816a366f-b67b-4679-a323-ab36289dc4b6-metrics-certs\") pod \"network-metrics-daemon-hm6lg\" (UID: \"816a366f-b67b-4679-a323-ab36289dc4b6\") " pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:30:43.059861 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:43.059800 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-original-pull-secret\") pod \"global-pull-secret-syncer-5s6pv\" (UID: \"2a102e1b-9b9b-45f3-9e7e-33794568bdb9\") " pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:30:43.060320 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:43.059914 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:30:43.060320 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:43.059974 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-original-pull-secret podName:2a102e1b-9b9b-45f3-9e7e-33794568bdb9 nodeName:}" failed. No retries permitted until 2026-09-16 13:30:45.059956668 +0000 UTC m=+6.104259625 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-original-pull-secret") pod "global-pull-secret-syncer-5s6pv" (UID: "2a102e1b-9b9b-45f3-9e7e-33794568bdb9") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:30:43.060722 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:43.060570 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:30:43.060722 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:43.060626 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/816a366f-b67b-4679-a323-ab36289dc4b6-metrics-certs podName:816a366f-b67b-4679-a323-ab36289dc4b6 nodeName:}" failed. No retries permitted until 2026-09-16 13:30:45.060607827 +0000 UTC m=+6.104910774 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/816a366f-b67b-4679-a323-ab36289dc4b6-metrics-certs") pod "network-metrics-daemon-hm6lg" (UID: "816a366f-b67b-4679-a323-ab36289dc4b6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:30:43.163436 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:43.161411 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzvrt\" (UniqueName: \"kubernetes.io/projected/205445dc-c385-470f-bc60-dfa12d107297-kube-api-access-nzvrt\") pod \"network-check-target-vg66w\" (UID: \"205445dc-c385-470f-bc60-dfa12d107297\") " pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:30:43.163436 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:43.161675 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:30:43.163436 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:43.161696 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:30:43.163436 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:43.161709 2562 projected.go:196] Error preparing data for projected volume kube-api-access-nzvrt for pod openshift-network-diagnostics/network-check-target-vg66w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:30:43.163436 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:43.161775 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/205445dc-c385-470f-bc60-dfa12d107297-kube-api-access-nzvrt podName:205445dc-c385-470f-bc60-dfa12d107297 nodeName:}" failed. No retries permitted until 2026-09-16 13:30:45.161756828 +0000 UTC m=+6.206059787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzvrt" (UniqueName: "kubernetes.io/projected/205445dc-c385-470f-bc60-dfa12d107297-kube-api-access-nzvrt") pod "network-check-target-vg66w" (UID: "205445dc-c385-470f-bc60-dfa12d107297") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:30:43.399679 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:43.399566 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 13:25:41 +0000 UTC" deadline="2028-06-26 15:48:25.692838902 +0000 UTC" Sep 16 13:30:43.399679 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:43.399627 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15578h17m42.293215725s" Sep 16 13:30:43.409618 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:43.409590 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:30:43.409753 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:43.409718 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hm6lg" podUID="816a366f-b67b-4679-a323-ab36289dc4b6" Sep 16 13:30:43.435642 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:43.435613 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-23.ec2.internal" event={"ID":"fc88820cac2d990b85f97c81f96e2d49","Type":"ContainerStarted","Data":"df339ab27acf940c92454e58c74baf134bee9bd8c4a78674eda7701b01fb1bd2"} Sep 16 13:30:44.407175 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:44.407138 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:30:44.407742 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:44.407259 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vg66w" podUID="205445dc-c385-470f-bc60-dfa12d107297" Sep 16 13:30:44.407742 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:44.407332 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:30:44.407742 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:44.407424 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s6pv" podUID="2a102e1b-9b9b-45f3-9e7e-33794568bdb9" Sep 16 13:30:44.454193 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:44.453656 2562 generic.go:356] "Generic (PLEG): container finished" podID="b7bf8d3fcfbbc3832a3c68332bac0716" containerID="28840da07457f5ad32b5b66ac4cccb02004448db99a156df53a3864c1b05d595" exitCode=0 Sep 16 13:30:44.454193 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:44.454083 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal" event={"ID":"b7bf8d3fcfbbc3832a3c68332bac0716","Type":"ContainerDied","Data":"28840da07457f5ad32b5b66ac4cccb02004448db99a156df53a3864c1b05d595"} Sep 16 13:30:44.474067 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:44.473995 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-23.ec2.internal" podStartSLOduration=4.473909222 podStartE2EDuration="4.473909222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:30:43.459780237 +0000 UTC m=+4.504083206" watchObservedRunningTime="2026-09-16 13:30:44.473909222 +0000 UTC m=+5.518212189" Sep 16 13:30:45.075438 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:45.075334 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-original-pull-secret\") pod \"global-pull-secret-syncer-5s6pv\" (UID: \"2a102e1b-9b9b-45f3-9e7e-33794568bdb9\") " pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:30:45.075438 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:45.075423 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/816a366f-b67b-4679-a323-ab36289dc4b6-metrics-certs\") pod \"network-metrics-daemon-hm6lg\" (UID: \"816a366f-b67b-4679-a323-ab36289dc4b6\") " pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:30:45.075649 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:45.075586 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:30:45.075649 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:45.075648 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/816a366f-b67b-4679-a323-ab36289dc4b6-metrics-certs podName:816a366f-b67b-4679-a323-ab36289dc4b6 nodeName:}" failed. No retries permitted until 2026-09-16 13:30:49.075628119 +0000 UTC m=+10.119931077 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/816a366f-b67b-4679-a323-ab36289dc4b6-metrics-certs") pod "network-metrics-daemon-hm6lg" (UID: "816a366f-b67b-4679-a323-ab36289dc4b6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:30:45.076047 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:45.076023 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:30:45.076176 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:45.076079 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-original-pull-secret podName:2a102e1b-9b9b-45f3-9e7e-33794568bdb9 nodeName:}" failed. No retries permitted until 2026-09-16 13:30:49.076065218 +0000 UTC m=+10.120368163 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-original-pull-secret") pod "global-pull-secret-syncer-5s6pv" (UID: "2a102e1b-9b9b-45f3-9e7e-33794568bdb9") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:30:45.175982 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:45.175948 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzvrt\" (UniqueName: \"kubernetes.io/projected/205445dc-c385-470f-bc60-dfa12d107297-kube-api-access-nzvrt\") pod \"network-check-target-vg66w\" (UID: \"205445dc-c385-470f-bc60-dfa12d107297\") " pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:30:45.176228 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:45.176208 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:30:45.176318 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:45.176233 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:30:45.176318 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:45.176247 2562 projected.go:196] Error preparing data for projected volume kube-api-access-nzvrt for pod openshift-network-diagnostics/network-check-target-vg66w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:30:45.176318 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:45.176299 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/205445dc-c385-470f-bc60-dfa12d107297-kube-api-access-nzvrt podName:205445dc-c385-470f-bc60-dfa12d107297 nodeName:}" failed. No retries permitted until 2026-09-16 13:30:49.17628367 +0000 UTC m=+10.220586631 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzvrt" (UniqueName: "kubernetes.io/projected/205445dc-c385-470f-bc60-dfa12d107297-kube-api-access-nzvrt") pod "network-check-target-vg66w" (UID: "205445dc-c385-470f-bc60-dfa12d107297") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:30:45.409013 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:45.408939 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:30:45.409452 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:45.409060 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hm6lg" podUID="816a366f-b67b-4679-a323-ab36289dc4b6" Sep 16 13:30:45.457467 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:45.457429 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal" event={"ID":"b7bf8d3fcfbbc3832a3c68332bac0716","Type":"ContainerStarted","Data":"b360bd6ac868edc7157908654efee61dfbab10a5ea34dc1e143c9328502a9565"} Sep 16 13:30:45.475716 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:45.475666 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal" podStartSLOduration=5.475645569 podStartE2EDuration="5.475645569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:30:45.475367488 +0000 UTC m=+6.519670453" watchObservedRunningTime="2026-09-16 13:30:45.475645569 +0000 UTC m=+6.519948538" Sep 16 13:30:46.407683 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:46.407649 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:30:46.407850 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:46.407777 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s6pv" podUID="2a102e1b-9b9b-45f3-9e7e-33794568bdb9" Sep 16 13:30:46.408143 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:46.408120 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:30:46.408236 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:46.408209 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vg66w" podUID="205445dc-c385-470f-bc60-dfa12d107297" Sep 16 13:30:47.407176 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:47.407145 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:30:47.407628 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:47.407267 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hm6lg" podUID="816a366f-b67b-4679-a323-ab36289dc4b6" Sep 16 13:30:48.407132 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:48.406848 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:30:48.407261 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:48.407191 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vg66w" podUID="205445dc-c385-470f-bc60-dfa12d107297" Sep 16 13:30:48.407597 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:48.406852 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:30:48.407648 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:48.407620 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s6pv" podUID="2a102e1b-9b9b-45f3-9e7e-33794568bdb9" Sep 16 13:30:49.111101 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:49.110915 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-original-pull-secret\") pod \"global-pull-secret-syncer-5s6pv\" (UID: \"2a102e1b-9b9b-45f3-9e7e-33794568bdb9\") " pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:30:49.111101 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:49.110963 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/816a366f-b67b-4679-a323-ab36289dc4b6-metrics-certs\") pod \"network-metrics-daemon-hm6lg\" (UID: \"816a366f-b67b-4679-a323-ab36289dc4b6\") " pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:30:49.111101 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:49.111073 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:30:49.111392 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:49.111135 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-original-pull-secret podName:2a102e1b-9b9b-45f3-9e7e-33794568bdb9 nodeName:}" failed. No retries permitted until 2026-09-16 13:30:57.111117939 +0000 UTC m=+18.155420885 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-original-pull-secret") pod "global-pull-secret-syncer-5s6pv" (UID: "2a102e1b-9b9b-45f3-9e7e-33794568bdb9") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:30:49.111527 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:49.111073 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:30:49.111527 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:49.111511 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/816a366f-b67b-4679-a323-ab36289dc4b6-metrics-certs podName:816a366f-b67b-4679-a323-ab36289dc4b6 nodeName:}" failed. No retries permitted until 2026-09-16 13:30:57.111496815 +0000 UTC m=+18.155799767 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/816a366f-b67b-4679-a323-ab36289dc4b6-metrics-certs") pod "network-metrics-daemon-hm6lg" (UID: "816a366f-b67b-4679-a323-ab36289dc4b6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:30:49.211870 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:49.211833 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzvrt\" (UniqueName: \"kubernetes.io/projected/205445dc-c385-470f-bc60-dfa12d107297-kube-api-access-nzvrt\") pod \"network-check-target-vg66w\" (UID: \"205445dc-c385-470f-bc60-dfa12d107297\") " pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:30:49.212063 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:49.212036 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:30:49.212201 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:49.212070 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:30:49.212201 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:49.212084 2562 projected.go:196] Error preparing data for projected volume kube-api-access-nzvrt for pod openshift-network-diagnostics/network-check-target-vg66w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:30:49.212201 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:49.212142 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/205445dc-c385-470f-bc60-dfa12d107297-kube-api-access-nzvrt podName:205445dc-c385-470f-bc60-dfa12d107297 nodeName:}" failed. No retries permitted until 2026-09-16 13:30:57.212122446 +0000 UTC m=+18.256425412 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzvrt" (UniqueName: "kubernetes.io/projected/205445dc-c385-470f-bc60-dfa12d107297-kube-api-access-nzvrt") pod "network-check-target-vg66w" (UID: "205445dc-c385-470f-bc60-dfa12d107297") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:30:49.411811 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:49.411734 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:30:49.412269 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:49.412249 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hm6lg" podUID="816a366f-b67b-4679-a323-ab36289dc4b6" Sep 16 13:30:50.407195 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:50.407144 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:30:50.407393 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:50.407266 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vg66w" podUID="205445dc-c385-470f-bc60-dfa12d107297" Sep 16 13:30:50.407514 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:50.407488 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:30:50.407621 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:50.407600 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s6pv" podUID="2a102e1b-9b9b-45f3-9e7e-33794568bdb9" Sep 16 13:30:51.408529 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:51.408497 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:30:51.408981 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:51.408635 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hm6lg" podUID="816a366f-b67b-4679-a323-ab36289dc4b6" Sep 16 13:30:52.407507 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:52.407460 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:30:52.407681 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:52.407460 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:30:52.407681 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:52.407569 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vg66w" podUID="205445dc-c385-470f-bc60-dfa12d107297" Sep 16 13:30:52.407681 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:52.407631 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s6pv" podUID="2a102e1b-9b9b-45f3-9e7e-33794568bdb9" Sep 16 13:30:53.407236 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:53.407163 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:30:53.407685 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:53.407303 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hm6lg" podUID="816a366f-b67b-4679-a323-ab36289dc4b6" Sep 16 13:30:54.407773 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:54.407747 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:30:54.408181 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:54.407754 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:30:54.408181 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:54.407846 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s6pv" podUID="2a102e1b-9b9b-45f3-9e7e-33794568bdb9" Sep 16 13:30:54.408181 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:54.407949 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vg66w" podUID="205445dc-c385-470f-bc60-dfa12d107297" Sep 16 13:30:55.407317 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:55.407285 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:30:55.407525 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:55.407431 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hm6lg" podUID="816a366f-b67b-4679-a323-ab36289dc4b6" Sep 16 13:30:56.406890 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:56.406855 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:30:56.407335 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:56.406976 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s6pv" podUID="2a102e1b-9b9b-45f3-9e7e-33794568bdb9" Sep 16 13:30:56.407335 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:56.407031 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:30:56.407335 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:56.407125 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vg66w" podUID="205445dc-c385-470f-bc60-dfa12d107297" Sep 16 13:30:57.169780 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:57.169743 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/816a366f-b67b-4679-a323-ab36289dc4b6-metrics-certs\") pod \"network-metrics-daemon-hm6lg\" (UID: \"816a366f-b67b-4679-a323-ab36289dc4b6\") " pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:30:57.169984 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:57.169832 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-original-pull-secret\") pod \"global-pull-secret-syncer-5s6pv\" (UID: \"2a102e1b-9b9b-45f3-9e7e-33794568bdb9\") " pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:30:57.169984 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:57.169916 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:30:57.169984 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:57.169936 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:30:57.169984 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:57.169980 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-original-pull-secret podName:2a102e1b-9b9b-45f3-9e7e-33794568bdb9 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:13.169968051 +0000 UTC m=+34.214271013 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-original-pull-secret") pod "global-pull-secret-syncer-5s6pv" (UID: "2a102e1b-9b9b-45f3-9e7e-33794568bdb9") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:30:57.170199 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:57.169994 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/816a366f-b67b-4679-a323-ab36289dc4b6-metrics-certs podName:816a366f-b67b-4679-a323-ab36289dc4b6 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:13.169987263 +0000 UTC m=+34.214290207 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/816a366f-b67b-4679-a323-ab36289dc4b6-metrics-certs") pod "network-metrics-daemon-hm6lg" (UID: "816a366f-b67b-4679-a323-ab36289dc4b6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:30:57.270262 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:57.270223 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzvrt\" (UniqueName: \"kubernetes.io/projected/205445dc-c385-470f-bc60-dfa12d107297-kube-api-access-nzvrt\") pod \"network-check-target-vg66w\" (UID: \"205445dc-c385-470f-bc60-dfa12d107297\") " pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:30:57.270426 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:57.270413 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:30:57.270473 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:57.270434 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:30:57.270473 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:57.270444 2562 projected.go:196] Error preparing data for projected volume kube-api-access-nzvrt for pod openshift-network-diagnostics/network-check-target-vg66w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:30:57.270532 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:57.270498 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/205445dc-c385-470f-bc60-dfa12d107297-kube-api-access-nzvrt podName:205445dc-c385-470f-bc60-dfa12d107297 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:13.270480972 +0000 UTC m=+34.314783919 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzvrt" (UniqueName: "kubernetes.io/projected/205445dc-c385-470f-bc60-dfa12d107297-kube-api-access-nzvrt") pod "network-check-target-vg66w" (UID: "205445dc-c385-470f-bc60-dfa12d107297") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:30:57.407762 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:57.407737 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:30:57.408208 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:57.407856 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hm6lg" podUID="816a366f-b67b-4679-a323-ab36289dc4b6" Sep 16 13:30:58.407212 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:58.407176 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:30:58.407417 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:58.407297 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vg66w" podUID="205445dc-c385-470f-bc60-dfa12d107297" Sep 16 13:30:58.407643 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:58.407186 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:30:58.407726 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:58.407715 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s6pv" podUID="2a102e1b-9b9b-45f3-9e7e-33794568bdb9" Sep 16 13:30:59.407731 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:30:59.407696 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:30:59.408172 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:30:59.407791 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hm6lg" podUID="816a366f-b67b-4679-a323-ab36289dc4b6" Sep 16 13:31:00.407795 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:00.407238 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:31:00.407795 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:00.407352 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s6pv" podUID="2a102e1b-9b9b-45f3-9e7e-33794568bdb9" Sep 16 13:31:00.407795 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:00.407688 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:31:00.407795 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:00.407770 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vg66w" podUID="205445dc-c385-470f-bc60-dfa12d107297" Sep 16 13:31:00.490600 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:00.490565 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8zxxf" event={"ID":"5695f6e6-c46d-4cd0-b4ab-9ab7c4edc9fc","Type":"ContainerStarted","Data":"5bec83e609e75367d49dd35c4466143dd733dae8c6d09a54b8ecf288b0091583"} Sep 16 13:31:01.407636 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:01.407486 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:31:01.407818 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:01.407725 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hm6lg" podUID="816a366f-b67b-4679-a323-ab36289dc4b6" Sep 16 13:31:01.493442 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:01.493406 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" event={"ID":"cdc69e4f-095b-453f-9114-8199d4a0477c","Type":"ContainerStarted","Data":"8f8bfc18f2dff561b85563dfaf8e57ed7fe077b1187351a76aa9f54f5754be51"} Sep 16 13:31:01.494914 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:01.494889 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dwt2h" event={"ID":"b9f0c124-231d-4677-8696-d7317828eb4f","Type":"ContainerStarted","Data":"ef2fbe63906b7a68045669a71543e4671d2ba0cf1eb7a91d54a61b7a11d111cc"} Sep 16 13:31:01.496175 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:01.496150 2562 generic.go:356] "Generic (PLEG): container finished" podID="870338a6-a8f2-46f8-a5e0-b1b17837057a" containerID="2117d555df5c7581f9c2db572c7a48ee1e24161cd24c3a1115d48e1d2e0d75a0" exitCode=0 Sep 16 13:31:01.496271 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:01.496229 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6bps" event={"ID":"870338a6-a8f2-46f8-a5e0-b1b17837057a","Type":"ContainerDied","Data":"2117d555df5c7581f9c2db572c7a48ee1e24161cd24c3a1115d48e1d2e0d75a0"} Sep 16 13:31:01.497718 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:01.497695 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t9qw9" event={"ID":"24582309-5bb1-4d3d-8a18-a1d7803f287a","Type":"ContainerStarted","Data":"50b067cce70aa4800c23a08119ab9a7c7cbe920afeb8566b119a15a1466e328a"} Sep 16 13:31:01.499187 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:01.499162 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mcc87" event={"ID":"e3cad145-80e4-4662-a1d8-e173c3cddf03","Type":"ContainerStarted","Data":"9bd4ab13405a996a8f56c5222657e262d219f94ddc0859f41845d0cb44f07d05"} Sep 16 13:31:01.501974 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:01.501953 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" event={"ID":"74bcf9e2-c283-4582-bbc2-a321c97c84b9","Type":"ContainerStarted","Data":"b75e77a0fcecd289565d59560ac21a504bf4d07cab66b1f2e85400cc35b53f1a"} Sep 16 13:31:01.502057 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:01.502016 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" event={"ID":"74bcf9e2-c283-4582-bbc2-a321c97c84b9","Type":"ContainerStarted","Data":"bd3cd7500970acc7dbad569b101bc1a353fb0e02725b3cad0591241cfb0f9a19"} Sep 16 13:31:01.502057 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:01.502052 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" event={"ID":"74bcf9e2-c283-4582-bbc2-a321c97c84b9","Type":"ContainerStarted","Data":"f35ac1a9a52487a2711acc7c2ed6bb5e02afc5e0f08bc819db091048bfc7dcf6"} Sep 16 13:31:01.502142 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:01.502079 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" event={"ID":"74bcf9e2-c283-4582-bbc2-a321c97c84b9","Type":"ContainerStarted","Data":"cd04cd4862ac30b6aa8a693ec7980474c650cfee82bf600f1e3cb23c7af0a7ad"} Sep 16 13:31:01.502142 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:01.502125 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" event={"ID":"74bcf9e2-c283-4582-bbc2-a321c97c84b9","Type":"ContainerStarted","Data":"8d2dd44c43b4ee9d59a76fbff8b2637abc01717ed597b6dd1b75113eece81837"} Sep 16 13:31:01.502226 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:01.502168 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" event={"ID":"74bcf9e2-c283-4582-bbc2-a321c97c84b9","Type":"ContainerStarted","Data":"a530080d0669593631efe7aa2310835bfb7da7872cf87216be622d9f3abba1d7"} Sep 16 13:31:01.503238 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:01.503217 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" event={"ID":"f7d94e4b-754b-40b0-85be-6f1293ec5f07","Type":"ContainerStarted","Data":"1c8295df2c096f9a0fd2a797b08f02a3dbe34744562b6de13bd4e009cbf8739f"} Sep 16 13:31:01.518696 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:01.518662 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8zxxf" podStartSLOduration=11.181390547 podStartE2EDuration="22.518646032s" totalImagesPullingTime="11.337255485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:00.510436211 +0000 UTC m=+21.554739178" watchObservedRunningTime="2026-09-16 13:31:01.518646032 +0000 UTC m=+22.562949019" Sep 16 13:31:01.540246 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:01.540214 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kmpb5" podStartSLOduration=3.890002361 podStartE2EDuration="22.540204815s" totalImagesPullingTime="18.650202454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:01.518488954 +0000 UTC m=+22.562791940" watchObservedRunningTime="2026-09-16 13:31:01.540204815 +0000 UTC m=+22.584507781" Sep 16 13:31:01.540646 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:01.540625 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dwt2h" podStartSLOduration=3.853976476 podStartE2EDuration="22.540619933s" totalImagesPullingTime="18.686643457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:01.539781144 +0000 UTC m=+22.584084108" watchObservedRunningTime="2026-09-16 13:31:01.540619933 +0000 UTC m=+22.584922897" Sep 16 13:31:01.617466 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:01.617425 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mcc87" podStartSLOduration=3.948804664 podStartE2EDuration="22.617414103s" totalImagesPullingTime="18.668609439s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:01.616250624 +0000 UTC m=+22.660553589" watchObservedRunningTime="2026-09-16 13:31:01.617414103 +0000 UTC m=+22.661717069" Sep 16 13:31:01.617793 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:01.617769 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-t9qw9" podStartSLOduration=3.986496848 podStartE2EDuration="22.617762213s" totalImagesPullingTime="18.631265365s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:01.595836084 +0000 UTC m=+22.640139050" watchObservedRunningTime="2026-09-16 13:31:01.617762213 +0000 UTC m=+22.662065234" Sep 16 13:31:02.061828 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:02.061690 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 16 13:31:02.407365 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:02.407296 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:31:02.407532 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:02.407296 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:31:02.407532 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:02.407430 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s6pv" podUID="2a102e1b-9b9b-45f3-9e7e-33794568bdb9" Sep 16 13:31:02.407532 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:02.407501 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vg66w" podUID="205445dc-c385-470f-bc60-dfa12d107297" Sep 16 13:31:02.409456 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:02.409370 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T13:31:02.061824904Z","UUID":"98f461a2-5085-4ab2-bbcd-e39cb0db28cb","Handler":null,"Name":"","Endpoint":""} Sep 16 13:31:02.411003 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:02.410983 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 16 13:31:02.411128 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:02.411014 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 16 13:31:02.507769 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:02.507731 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" event={"ID":"f7d94e4b-754b-40b0-85be-6f1293ec5f07","Type":"ContainerStarted","Data":"c6ed799720e9ae43dd514aee17f51d06e242a5b1aafac45b2bc95a7f04bccd67"} Sep 16 13:31:02.509463 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:02.509438 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hwrln" event={"ID":"2afeee9c-fe7c-4808-a71c-239a0c68a6a5","Type":"ContainerStarted","Data":"4932f8480d5de7d8aeb4ea2067902d59cfc387cf94addf8fa2b31c54d4e4b0a9"} Sep 16 13:31:02.526256 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:02.526221 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hwrln" podStartSLOduration=4.868492221 podStartE2EDuration="23.526209905s" totalImagesPullingTime="18.657717684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:02.525998629 +0000 UTC m=+23.570301613" watchObservedRunningTime="2026-09-16 13:31:02.526209905 +0000 UTC m=+23.570512872" Sep 16 13:31:03.407719 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:03.407606 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:31:03.407900 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:03.407736 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hm6lg" podUID="816a366f-b67b-4679-a323-ab36289dc4b6" Sep 16 13:31:03.514403 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:03.514362 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" event={"ID":"74bcf9e2-c283-4582-bbc2-a321c97c84b9","Type":"ContainerStarted","Data":"21a8e1b45f0188b5ded5153dec0c09608a38524a55ed129d6fa0c89213f95ad7"} Sep 16 13:31:04.053408 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:04.053360 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8zxxf" Sep 16 13:31:04.053964 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:04.053945 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8zxxf" Sep 16 13:31:04.407476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:04.407402 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:31:04.407632 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:04.407509 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s6pv" podUID="2a102e1b-9b9b-45f3-9e7e-33794568bdb9" Sep 16 13:31:04.407632 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:04.407572 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:31:04.407734 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:04.407682 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vg66w" podUID="205445dc-c385-470f-bc60-dfa12d107297" Sep 16 13:31:04.518728 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:04.518690 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" event={"ID":"f7d94e4b-754b-40b0-85be-6f1293ec5f07","Type":"ContainerStarted","Data":"1beb5721e2f09d3f1613148bf43baf6ede626611d3e7a6b0dc6d7e26a1c34c6c"} Sep 16 13:31:04.519099 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:04.518861 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8zxxf" Sep 16 13:31:04.519352 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:04.519333 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8zxxf" Sep 16 13:31:04.563894 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:04.563844 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p5r5v" podStartSLOduration=4.175947667 podStartE2EDuration="25.563829503s" totalImagesPullingTime="21.387881836s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:30:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:04.545258564 +0000 UTC m=+25.589561530" watchObservedRunningTime="2026-09-16 13:31:04.563829503 +0000 UTC m=+25.608132469" Sep 16 13:31:05.410334 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:05.410152 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:31:05.410504 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:05.410454 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hm6lg" podUID="816a366f-b67b-4679-a323-ab36289dc4b6" Sep 16 13:31:05.524584 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:05.524504 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" event={"ID":"74bcf9e2-c283-4582-bbc2-a321c97c84b9","Type":"ContainerStarted","Data":"cab873cf060026e03a14e746bf17a97742e953e5d0df727659412fd1da096a65"} Sep 16 13:31:05.566156 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:05.566083 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" podStartSLOduration=7.495861979 podStartE2EDuration="26.566068482s" totalImagesPullingTime="19.070206503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:05.565372546 +0000 UTC m=+26.609675532" watchObservedRunningTime="2026-09-16 13:31:05.566068482 +0000 UTC m=+26.610371448" Sep 16 13:31:06.407512 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:06.407481 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:31:06.407667 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:06.407483 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:31:06.407667 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:06.407585 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s6pv" podUID="2a102e1b-9b9b-45f3-9e7e-33794568bdb9" Sep 16 13:31:06.407742 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:06.407674 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vg66w" podUID="205445dc-c385-470f-bc60-dfa12d107297" Sep 16 13:31:06.529258 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:06.529069 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:31:06.529698 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:06.529546 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:31:06.529698 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:06.529613 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:31:06.550500 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:06.548014 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:31:06.555883 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:06.555565 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:31:07.237950 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:07.237918 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5s6pv"] Sep 16 13:31:07.238148 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:07.238031 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:31:07.238148 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:07.238130 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s6pv" podUID="2a102e1b-9b9b-45f3-9e7e-33794568bdb9" Sep 16 13:31:07.241026 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:07.240999 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hm6lg"] Sep 16 13:31:07.241140 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:07.241099 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:31:07.241237 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:07.241201 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hm6lg" podUID="816a366f-b67b-4679-a323-ab36289dc4b6" Sep 16 13:31:07.259224 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:07.259203 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vg66w"] Sep 16 13:31:07.259309 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:07.259273 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:31:07.259359 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:07.259343 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vg66w" podUID="205445dc-c385-470f-bc60-dfa12d107297" Sep 16 13:31:08.533481 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:08.533446 2562 generic.go:356] "Generic (PLEG): container finished" podID="870338a6-a8f2-46f8-a5e0-b1b17837057a" containerID="a84e4947b5bc00146ffe7b4d5c6388e9e10b32e87ebe46f8cc506c5b2a10decd" exitCode=0 Sep 16 13:31:08.534116 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:08.533526 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6bps" event={"ID":"870338a6-a8f2-46f8-a5e0-b1b17837057a","Type":"ContainerDied","Data":"a84e4947b5bc00146ffe7b4d5c6388e9e10b32e87ebe46f8cc506c5b2a10decd"} Sep 16 13:31:09.410013 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:09.409853 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:31:09.410155 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:09.409912 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:31:09.410155 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:09.410079 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vg66w" podUID="205445dc-c385-470f-bc60-dfa12d107297" Sep 16 13:31:09.410155 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:09.409912 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:31:09.410279 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:09.410172 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hm6lg" podUID="816a366f-b67b-4679-a323-ab36289dc4b6" Sep 16 13:31:09.410279 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:09.410221 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s6pv" podUID="2a102e1b-9b9b-45f3-9e7e-33794568bdb9" Sep 16 13:31:10.538789 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:10.538754 2562 generic.go:356] "Generic (PLEG): container finished" podID="870338a6-a8f2-46f8-a5e0-b1b17837057a" containerID="e1e69dc91196721496f42ac07642046770bfd041984c8705be1f08bd4aed053a" exitCode=0 Sep 16 13:31:10.539130 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:10.538817 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6bps" event={"ID":"870338a6-a8f2-46f8-a5e0-b1b17837057a","Type":"ContainerDied","Data":"e1e69dc91196721496f42ac07642046770bfd041984c8705be1f08bd4aed053a"} Sep 16 13:31:11.411152 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:11.411124 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:31:11.411299 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:11.411216 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:31:11.411299 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:11.411228 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s6pv" podUID="2a102e1b-9b9b-45f3-9e7e-33794568bdb9" Sep 16 13:31:11.411299 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:11.411250 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:31:11.411450 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:11.411330 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vg66w" podUID="205445dc-c385-470f-bc60-dfa12d107297" Sep 16 13:31:11.411487 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:11.411447 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hm6lg" podUID="816a366f-b67b-4679-a323-ab36289dc4b6" Sep 16 13:31:12.548257 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:12.548220 2562 generic.go:356] "Generic (PLEG): container finished" podID="870338a6-a8f2-46f8-a5e0-b1b17837057a" containerID="d4058ae16801c87593a695aea737a2e38175bff44172f14ddef0149588b5438c" exitCode=0 Sep 16 13:31:12.548677 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:12.548281 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6bps" event={"ID":"870338a6-a8f2-46f8-a5e0-b1b17837057a","Type":"ContainerDied","Data":"d4058ae16801c87593a695aea737a2e38175bff44172f14ddef0149588b5438c"} Sep 16 13:31:13.198658 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.198639 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/816a366f-b67b-4679-a323-ab36289dc4b6-metrics-certs\") pod \"network-metrics-daemon-hm6lg\" (UID: \"816a366f-b67b-4679-a323-ab36289dc4b6\") " pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:31:13.198797 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.198690 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-original-pull-secret\") pod \"global-pull-secret-syncer-5s6pv\" (UID: \"2a102e1b-9b9b-45f3-9e7e-33794568bdb9\") " pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:31:13.198797 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:13.198776 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 13:31:13.198797 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:13.198783 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:31:13.198916 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:13.198828 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-original-pull-secret podName:2a102e1b-9b9b-45f3-9e7e-33794568bdb9 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:45.198812581 +0000 UTC m=+66.243115526 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-original-pull-secret") pod "global-pull-secret-syncer-5s6pv" (UID: "2a102e1b-9b9b-45f3-9e7e-33794568bdb9") : object "kube-system"/"original-pull-secret" not registered Sep 16 13:31:13.198916 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:13.198843 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/816a366f-b67b-4679-a323-ab36289dc4b6-metrics-certs podName:816a366f-b67b-4679-a323-ab36289dc4b6 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:45.198835706 +0000 UTC m=+66.243138650 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/816a366f-b67b-4679-a323-ab36289dc4b6-metrics-certs") pod "network-metrics-daemon-hm6lg" (UID: "816a366f-b67b-4679-a323-ab36289dc4b6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 13:31:13.246192 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.246170 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeReady" Sep 16 13:31:13.246288 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.246262 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 16 13:31:13.299066 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.299014 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzvrt\" (UniqueName: \"kubernetes.io/projected/205445dc-c385-470f-bc60-dfa12d107297-kube-api-access-nzvrt\") pod \"network-check-target-vg66w\" (UID: \"205445dc-c385-470f-bc60-dfa12d107297\") " pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:31:13.299155 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:13.299122 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 13:31:13.299155 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:13.299135 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 13:31:13.299155 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:13.299142 2562 projected.go:196] Error preparing data for projected volume kube-api-access-nzvrt for pod openshift-network-diagnostics/network-check-target-vg66w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:31:13.299245 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:13.299185 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/205445dc-c385-470f-bc60-dfa12d107297-kube-api-access-nzvrt podName:205445dc-c385-470f-bc60-dfa12d107297 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:45.299174386 +0000 UTC m=+66.343477331 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzvrt" (UniqueName: "kubernetes.io/projected/205445dc-c385-470f-bc60-dfa12d107297-kube-api-access-nzvrt") pod "network-check-target-vg66w" (UID: "205445dc-c385-470f-bc60-dfa12d107297") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 13:31:13.308998 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.308979 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6449dbdc57-jc64r"] Sep 16 13:31:13.333225 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.333192 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mxnjp"] Sep 16 13:31:13.336732 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.335455 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:13.342249 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.342232 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s4fpm\"" Sep 16 13:31:13.342520 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.342505 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 16 13:31:13.342901 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.342884 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 16 13:31:13.343009 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.342889 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 16 13:31:13.348763 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.348743 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sfzx5"] Sep 16 13:31:13.348879 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.348860 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mxnjp" Sep 16 13:31:13.349687 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.349664 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 16 13:31:13.351812 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.351604 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 16 13:31:13.351812 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.351648 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 16 13:31:13.352294 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.352131 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t4m6m\"" Sep 16 13:31:13.366801 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.366780 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6449dbdc57-jc64r"] Sep 16 13:31:13.366879 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.366808 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sfzx5"] Sep 16 13:31:13.366879 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.366820 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mxnjp"] Sep 16 13:31:13.366879 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.366870 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sfzx5" Sep 16 13:31:13.369815 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.369796 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 16 13:31:13.369891 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.369796 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 16 13:31:13.369891 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.369839 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 16 13:31:13.369997 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.369899 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7jfkh\"" Sep 16 13:31:13.407827 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.407806 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:31:13.407930 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.407914 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:31:13.407966 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.407946 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:31:13.412362 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.412344 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4nxp8\"" Sep 16 13:31:13.412470 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.412438 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 13:31:13.412470 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.412448 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 13:31:13.412470 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.412414 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 13:31:13.413061 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.413048 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4n8lp\"" Sep 16 13:31:13.413281 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.413269 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 13:31:13.474518 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.474497 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8zxxf_5695f6e6-c46d-4cd0-b4ab-9ab7c4edc9fc/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:13.500462 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.500441 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5ab9c39-b9a9-42e1-8175-45c414ec2de9-cert\") pod \"ingress-canary-sfzx5\" (UID: \"d5ab9c39-b9a9-42e1-8175-45c414ec2de9\") " pod="openshift-ingress-canary/ingress-canary-sfzx5" Sep 16 13:31:13.500535 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.500467 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zglpk\" (UniqueName: \"kubernetes.io/projected/d5ab9c39-b9a9-42e1-8175-45c414ec2de9-kube-api-access-zglpk\") pod \"ingress-canary-sfzx5\" (UID: \"d5ab9c39-b9a9-42e1-8175-45c414ec2de9\") " pod="openshift-ingress-canary/ingress-canary-sfzx5" Sep 16 13:31:13.500535 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.500485 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/390ee01e-b850-45d2-b21d-c6b002a1c0ff-config-volume\") pod \"dns-default-mxnjp\" (UID: \"390ee01e-b850-45d2-b21d-c6b002a1c0ff\") " pod="openshift-dns/dns-default-mxnjp" Sep 16 13:31:13.500535 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.500502 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw5b5\" (UniqueName: \"kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-kube-api-access-hw5b5\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:13.500638 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.500569 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-certificates\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:13.500638 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.500599 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/390ee01e-b850-45d2-b21d-c6b002a1c0ff-metrics-tls\") pod \"dns-default-mxnjp\" (UID: \"390ee01e-b850-45d2-b21d-c6b002a1c0ff\") " pod="openshift-dns/dns-default-mxnjp" Sep 16 13:31:13.500638 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.500618 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwpb5\" (UniqueName: \"kubernetes.io/projected/390ee01e-b850-45d2-b21d-c6b002a1c0ff-kube-api-access-lwpb5\") pod \"dns-default-mxnjp\" (UID: \"390ee01e-b850-45d2-b21d-c6b002a1c0ff\") " pod="openshift-dns/dns-default-mxnjp" Sep 16 13:31:13.500738 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.500656 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-tls\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:13.500738 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.500680 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-ca-trust-extracted\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:13.500738 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.500694 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-installation-pull-secrets\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:13.500738 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.500708 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-bound-sa-token\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:13.500738 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.500729 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/390ee01e-b850-45d2-b21d-c6b002a1c0ff-tmp-dir\") pod \"dns-default-mxnjp\" (UID: \"390ee01e-b850-45d2-b21d-c6b002a1c0ff\") " pod="openshift-dns/dns-default-mxnjp" Sep 16 13:31:13.500891 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.500749 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-image-registry-private-configuration\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:13.500891 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.500771 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-trusted-ca\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:13.593033 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.592968 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8zxxf_5695f6e6-c46d-4cd0-b4ab-9ab7c4edc9fc/konnectivity-agent/0.log" limit=1024 Sep 16 13:31:13.601934 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.601913 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/390ee01e-b850-45d2-b21d-c6b002a1c0ff-metrics-tls\") pod \"dns-default-mxnjp\" (UID: \"390ee01e-b850-45d2-b21d-c6b002a1c0ff\") " pod="openshift-dns/dns-default-mxnjp" Sep 16 13:31:13.601989 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.601939 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwpb5\" (UniqueName: \"kubernetes.io/projected/390ee01e-b850-45d2-b21d-c6b002a1c0ff-kube-api-access-lwpb5\") pod \"dns-default-mxnjp\" (UID: \"390ee01e-b850-45d2-b21d-c6b002a1c0ff\") " pod="openshift-dns/dns-default-mxnjp" Sep 16 13:31:13.601989 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.601971 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-tls\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:13.602056 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:13.602041 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:31:13.602111 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.602074 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-ca-trust-extracted\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:13.602111 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:13.602082 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/390ee01e-b850-45d2-b21d-c6b002a1c0ff-metrics-tls podName:390ee01e-b850-45d2-b21d-c6b002a1c0ff nodeName:}" failed. No retries permitted until 2026-09-16 13:31:14.102069072 +0000 UTC m=+35.146372016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/390ee01e-b850-45d2-b21d-c6b002a1c0ff-metrics-tls") pod "dns-default-mxnjp" (UID: "390ee01e-b850-45d2-b21d-c6b002a1c0ff") : secret "dns-default-metrics-tls" not found Sep 16 13:31:13.602219 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.602119 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-installation-pull-secrets\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:13.602219 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.602147 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-bound-sa-token\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:13.602219 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:13.602164 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 13:31:13.602219 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:13.602181 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6449dbdc57-jc64r: secret "image-registry-tls" not found Sep 16 13:31:13.602476 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:13.602242 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-tls podName:58c0f9ae-bb5d-4621-b804-29b5a3a88f6c nodeName:}" failed. No retries permitted until 2026-09-16 13:31:14.102225093 +0000 UTC m=+35.146528081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-tls") pod "image-registry-6449dbdc57-jc64r" (UID: "58c0f9ae-bb5d-4621-b804-29b5a3a88f6c") : secret "image-registry-tls" not found Sep 16 13:31:13.602476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.602179 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/390ee01e-b850-45d2-b21d-c6b002a1c0ff-tmp-dir\") pod \"dns-default-mxnjp\" (UID: \"390ee01e-b850-45d2-b21d-c6b002a1c0ff\") " pod="openshift-dns/dns-default-mxnjp" Sep 16 13:31:13.602476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.602291 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-image-registry-private-configuration\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:13.602476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.602325 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-trusted-ca\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:13.602476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.602369 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5ab9c39-b9a9-42e1-8175-45c414ec2de9-cert\") pod \"ingress-canary-sfzx5\" (UID: \"d5ab9c39-b9a9-42e1-8175-45c414ec2de9\") " pod="openshift-ingress-canary/ingress-canary-sfzx5" Sep 16 13:31:13.602476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.602420 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zglpk\" (UniqueName: \"kubernetes.io/projected/d5ab9c39-b9a9-42e1-8175-45c414ec2de9-kube-api-access-zglpk\") pod \"ingress-canary-sfzx5\" (UID: \"d5ab9c39-b9a9-42e1-8175-45c414ec2de9\") " pod="openshift-ingress-canary/ingress-canary-sfzx5" Sep 16 13:31:13.602476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.602447 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/390ee01e-b850-45d2-b21d-c6b002a1c0ff-config-volume\") pod \"dns-default-mxnjp\" (UID: \"390ee01e-b850-45d2-b21d-c6b002a1c0ff\") " pod="openshift-dns/dns-default-mxnjp" Sep 16 13:31:13.602476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.602472 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-ca-trust-extracted\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:13.602866 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.602472 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/390ee01e-b850-45d2-b21d-c6b002a1c0ff-tmp-dir\") pod \"dns-default-mxnjp\" (UID: \"390ee01e-b850-45d2-b21d-c6b002a1c0ff\") " pod="openshift-dns/dns-default-mxnjp" Sep 16 13:31:13.602866 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.602535 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw5b5\" (UniqueName: \"kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-kube-api-access-hw5b5\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:13.602866 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.602603 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-certificates\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:13.603160 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.603132 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-certificates\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:13.603160 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.603151 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/390ee01e-b850-45d2-b21d-c6b002a1c0ff-config-volume\") pod \"dns-default-mxnjp\" (UID: \"390ee01e-b850-45d2-b21d-c6b002a1c0ff\") " pod="openshift-dns/dns-default-mxnjp" Sep 16 13:31:13.603903 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.603853 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-trusted-ca\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:13.606219 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.606193 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-installation-pull-secrets\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:13.606219 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.606209 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-image-registry-private-configuration\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:13.606331 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.606247 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5ab9c39-b9a9-42e1-8175-45c414ec2de9-cert\") pod \"ingress-canary-sfzx5\" (UID: \"d5ab9c39-b9a9-42e1-8175-45c414ec2de9\") " pod="openshift-ingress-canary/ingress-canary-sfzx5" Sep 16 13:31:13.613172 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.613148 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwpb5\" (UniqueName: \"kubernetes.io/projected/390ee01e-b850-45d2-b21d-c6b002a1c0ff-kube-api-access-lwpb5\") pod \"dns-default-mxnjp\" (UID: \"390ee01e-b850-45d2-b21d-c6b002a1c0ff\") " pod="openshift-dns/dns-default-mxnjp" Sep 16 13:31:13.614437 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.614419 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-bound-sa-token\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:13.614506 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.614482 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zglpk\" (UniqueName: \"kubernetes.io/projected/d5ab9c39-b9a9-42e1-8175-45c414ec2de9-kube-api-access-zglpk\") pod \"ingress-canary-sfzx5\" (UID: \"d5ab9c39-b9a9-42e1-8175-45c414ec2de9\") " pod="openshift-ingress-canary/ingress-canary-sfzx5" Sep 16 13:31:13.614898 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.614882 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw5b5\" (UniqueName: \"kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-kube-api-access-hw5b5\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:13.674154 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:13.674132 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sfzx5" Sep 16 13:31:14.000721 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:14.000559 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sfzx5"] Sep 16 13:31:14.003853 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:31:14.003822 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ab9c39_b9a9_42e1_8175_45c414ec2de9.slice/crio-be73d6e4cd81fd240f9a46050833ffcd2d0693800e8a5ca794fbb44090c42601 WatchSource:0}: Error finding container be73d6e4cd81fd240f9a46050833ffcd2d0693800e8a5ca794fbb44090c42601: Status 404 returned error can't find the container with id be73d6e4cd81fd240f9a46050833ffcd2d0693800e8a5ca794fbb44090c42601 Sep 16 13:31:14.106302 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:14.106267 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/390ee01e-b850-45d2-b21d-c6b002a1c0ff-metrics-tls\") pod \"dns-default-mxnjp\" (UID: \"390ee01e-b850-45d2-b21d-c6b002a1c0ff\") " pod="openshift-dns/dns-default-mxnjp" Sep 16 13:31:14.106454 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:14.106367 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-tls\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:14.106585 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:14.106568 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 13:31:14.106635 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:14.106587 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6449dbdc57-jc64r: secret "image-registry-tls" not found Sep 16 13:31:14.106686 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:14.106656 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-tls podName:58c0f9ae-bb5d-4621-b804-29b5a3a88f6c nodeName:}" failed. No retries permitted until 2026-09-16 13:31:15.106639177 +0000 UTC m=+36.150942151 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-tls") pod "image-registry-6449dbdc57-jc64r" (UID: "58c0f9ae-bb5d-4621-b804-29b5a3a88f6c") : secret "image-registry-tls" not found Sep 16 13:31:14.106758 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:14.106746 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:31:14.106807 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:14.106788 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/390ee01e-b850-45d2-b21d-c6b002a1c0ff-metrics-tls podName:390ee01e-b850-45d2-b21d-c6b002a1c0ff nodeName:}" failed. No retries permitted until 2026-09-16 13:31:15.106777551 +0000 UTC m=+36.151080512 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/390ee01e-b850-45d2-b21d-c6b002a1c0ff-metrics-tls") pod "dns-default-mxnjp" (UID: "390ee01e-b850-45d2-b21d-c6b002a1c0ff") : secret "dns-default-metrics-tls" not found Sep 16 13:31:14.554013 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:14.553974 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sfzx5" event={"ID":"d5ab9c39-b9a9-42e1-8175-45c414ec2de9","Type":"ContainerStarted","Data":"be73d6e4cd81fd240f9a46050833ffcd2d0693800e8a5ca794fbb44090c42601"} Sep 16 13:31:15.114170 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:15.114130 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/390ee01e-b850-45d2-b21d-c6b002a1c0ff-metrics-tls\") pod \"dns-default-mxnjp\" (UID: \"390ee01e-b850-45d2-b21d-c6b002a1c0ff\") " pod="openshift-dns/dns-default-mxnjp" Sep 16 13:31:15.114772 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:15.114199 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-tls\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:15.114772 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:15.114310 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:31:15.114772 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:15.114315 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 13:31:15.114772 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:15.114414 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/390ee01e-b850-45d2-b21d-c6b002a1c0ff-metrics-tls podName:390ee01e-b850-45d2-b21d-c6b002a1c0ff nodeName:}" failed. No retries permitted until 2026-09-16 13:31:17.114394628 +0000 UTC m=+38.158697588 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/390ee01e-b850-45d2-b21d-c6b002a1c0ff-metrics-tls") pod "dns-default-mxnjp" (UID: "390ee01e-b850-45d2-b21d-c6b002a1c0ff") : secret "dns-default-metrics-tls" not found Sep 16 13:31:15.114772 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:15.114417 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6449dbdc57-jc64r: secret "image-registry-tls" not found Sep 16 13:31:15.114772 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:15.114481 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-tls podName:58c0f9ae-bb5d-4621-b804-29b5a3a88f6c nodeName:}" failed. No retries permitted until 2026-09-16 13:31:17.114465396 +0000 UTC m=+38.158768346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-tls") pod "image-registry-6449dbdc57-jc64r" (UID: "58c0f9ae-bb5d-4621-b804-29b5a3a88f6c") : secret "image-registry-tls" not found Sep 16 13:31:17.130286 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:17.130245 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/390ee01e-b850-45d2-b21d-c6b002a1c0ff-metrics-tls\") pod \"dns-default-mxnjp\" (UID: \"390ee01e-b850-45d2-b21d-c6b002a1c0ff\") " pod="openshift-dns/dns-default-mxnjp" Sep 16 13:31:17.130893 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:17.130294 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-tls\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:17.130893 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:17.130442 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 13:31:17.130893 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:17.130434 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:31:17.130893 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:17.130459 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6449dbdc57-jc64r: secret "image-registry-tls" not found Sep 16 13:31:17.130893 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:17.130524 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/390ee01e-b850-45d2-b21d-c6b002a1c0ff-metrics-tls podName:390ee01e-b850-45d2-b21d-c6b002a1c0ff nodeName:}" failed. No retries permitted until 2026-09-16 13:31:21.130502545 +0000 UTC m=+42.174805498 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/390ee01e-b850-45d2-b21d-c6b002a1c0ff-metrics-tls") pod "dns-default-mxnjp" (UID: "390ee01e-b850-45d2-b21d-c6b002a1c0ff") : secret "dns-default-metrics-tls" not found Sep 16 13:31:17.130893 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:17.130542 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-tls podName:58c0f9ae-bb5d-4621-b804-29b5a3a88f6c nodeName:}" failed. No retries permitted until 2026-09-16 13:31:21.130533558 +0000 UTC m=+42.174836504 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-tls") pod "image-registry-6449dbdc57-jc64r" (UID: "58c0f9ae-bb5d-4621-b804-29b5a3a88f6c") : secret "image-registry-tls" not found Sep 16 13:31:17.562900 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:17.562861 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sfzx5" event={"ID":"d5ab9c39-b9a9-42e1-8175-45c414ec2de9","Type":"ContainerStarted","Data":"bb524127b06be2d1f87e5d4cc7f17420e782e4380bd396b140551bb8d5d22b2b"} Sep 16 13:31:17.588135 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:17.588081 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sfzx5" podStartSLOduration=1.935399761 podStartE2EDuration="4.588064262s" totalImagesPullingTime="2.652664501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:31:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:17.587549742 +0000 UTC m=+38.631852727" watchObservedRunningTime="2026-09-16 13:31:17.588064262 +0000 UTC m=+38.632367228" Sep 16 13:31:19.570690 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:19.570515 2562 generic.go:356] "Generic (PLEG): container finished" podID="870338a6-a8f2-46f8-a5e0-b1b17837057a" containerID="b3eed90bc0bcd94cd4fca92a704a516017117d7b2753d577960b16f2888c3d74" exitCode=0 Sep 16 13:31:19.571023 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:19.570575 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6bps" event={"ID":"870338a6-a8f2-46f8-a5e0-b1b17837057a","Type":"ContainerDied","Data":"b3eed90bc0bcd94cd4fca92a704a516017117d7b2753d577960b16f2888c3d74"} Sep 16 13:31:20.575801 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:20.575772 2562 generic.go:356] "Generic (PLEG): container finished" podID="870338a6-a8f2-46f8-a5e0-b1b17837057a" containerID="aa46562caf544f1f56b35e1bf1040b4ad1b01a04eed5e55302c50a9674e8135c" exitCode=0 Sep 16 13:31:20.576147 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:20.575825 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6bps" event={"ID":"870338a6-a8f2-46f8-a5e0-b1b17837057a","Type":"ContainerDied","Data":"aa46562caf544f1f56b35e1bf1040b4ad1b01a04eed5e55302c50a9674e8135c"} Sep 16 13:31:21.162410 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.162348 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/390ee01e-b850-45d2-b21d-c6b002a1c0ff-metrics-tls\") pod \"dns-default-mxnjp\" (UID: \"390ee01e-b850-45d2-b21d-c6b002a1c0ff\") " pod="openshift-dns/dns-default-mxnjp" Sep 16 13:31:21.162569 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.162418 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-tls\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:21.162569 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:21.162494 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:31:21.162569 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:21.162515 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 13:31:21.162569 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:21.162533 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6449dbdc57-jc64r: secret "image-registry-tls" not found Sep 16 13:31:21.162569 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:21.162555 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/390ee01e-b850-45d2-b21d-c6b002a1c0ff-metrics-tls podName:390ee01e-b850-45d2-b21d-c6b002a1c0ff nodeName:}" failed. No retries permitted until 2026-09-16 13:31:29.162540132 +0000 UTC m=+50.206843076 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/390ee01e-b850-45d2-b21d-c6b002a1c0ff-metrics-tls") pod "dns-default-mxnjp" (UID: "390ee01e-b850-45d2-b21d-c6b002a1c0ff") : secret "dns-default-metrics-tls" not found Sep 16 13:31:21.162761 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:21.162580 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-tls podName:58c0f9ae-bb5d-4621-b804-29b5a3a88f6c nodeName:}" failed. No retries permitted until 2026-09-16 13:31:29.162564462 +0000 UTC m=+50.206867427 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-tls") pod "image-registry-6449dbdc57-jc64r" (UID: "58c0f9ae-bb5d-4621-b804-29b5a3a88f6c") : secret "image-registry-tls" not found Sep 16 13:31:21.261987 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.261959 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-vdgqh"] Sep 16 13:31:21.283704 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.283685 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-vdgqh"] Sep 16 13:31:21.283813 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.283770 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-vdgqh" Sep 16 13:31:21.286901 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.286880 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 16 13:31:21.287137 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.287118 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 16 13:31:21.287137 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.287133 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 16 13:31:21.287259 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.287236 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 16 13:31:21.288117 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.288101 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dn6jf\"" Sep 16 13:31:21.304059 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.304037 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 16 13:31:21.368758 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.368737 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-pw5cd"] Sep 16 13:31:21.381586 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.381568 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s2cmm"] Sep 16 13:31:21.381692 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.381676 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pw5cd" Sep 16 13:31:21.385992 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.385955 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 13:31:21.386089 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.386024 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tg274\"" Sep 16 13:31:21.386156 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.386095 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 16 13:31:21.386393 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.386357 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 13:31:21.386473 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.386368 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 16 13:31:21.395532 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.395511 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q995f"] Sep 16 13:31:21.395656 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.395639 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s2cmm" Sep 16 13:31:21.398369 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.398350 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 16 13:31:21.398618 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.398595 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gx5mx\"" Sep 16 13:31:21.398786 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.398617 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:31:21.398786 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.398632 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 16 13:31:21.411638 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.411620 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-zb868"] Sep 16 13:31:21.411734 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.411720 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q995f" Sep 16 13:31:21.414294 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.414249 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jvdcb\"" Sep 16 13:31:21.414294 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.414253 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:31:21.414434 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.414301 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 16 13:31:21.421704 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.421689 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-858dfcf476-rkhcs"] Sep 16 13:31:21.421815 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.421800 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-zb868" Sep 16 13:31:21.424497 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.424477 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 16 13:31:21.424627 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.424530 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:31:21.424724 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.424705 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 16 13:31:21.424724 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.424705 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-x5z7l\"" Sep 16 13:31:21.424840 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.424797 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 16 13:31:21.438477 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.438460 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-pw5cd"] Sep 16 13:31:21.438572 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.438557 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:21.440778 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.440759 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 16 13:31:21.440955 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.440934 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 16 13:31:21.441262 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.441231 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 16 13:31:21.441262 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.441251 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 16 13:31:21.441449 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.441321 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6lms4\"" Sep 16 13:31:21.441449 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.441333 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 16 13:31:21.441607 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.441593 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 16 13:31:21.442155 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.442141 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s2cmm"] Sep 16 13:31:21.442202 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.442175 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-zb868"] Sep 16 13:31:21.442202 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.442189 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-858dfcf476-rkhcs"] Sep 16 13:31:21.442202 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.442203 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q995f"] Sep 16 13:31:21.465332 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.465307 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3-service-ca-bundle\") pod \"insights-operator-5844b856db-vdgqh\" (UID: \"647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3\") " pod="openshift-insights/insights-operator-5844b856db-vdgqh" Sep 16 13:31:21.465447 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.465337 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3-serving-cert\") pod \"insights-operator-5844b856db-vdgqh\" (UID: \"647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3\") " pod="openshift-insights/insights-operator-5844b856db-vdgqh" Sep 16 13:31:21.465447 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.465364 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3-snapshots\") pod \"insights-operator-5844b856db-vdgqh\" (UID: \"647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3\") " pod="openshift-insights/insights-operator-5844b856db-vdgqh" Sep 16 13:31:21.465447 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.465440 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl8jb\" (UniqueName: \"kubernetes.io/projected/647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3-kube-api-access-hl8jb\") pod \"insights-operator-5844b856db-vdgqh\" (UID: \"647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3\") " pod="openshift-insights/insights-operator-5844b856db-vdgqh" Sep 16 13:31:21.465612 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.465510 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3-tmp\") pod \"insights-operator-5844b856db-vdgqh\" (UID: \"647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3\") " pod="openshift-insights/insights-operator-5844b856db-vdgqh" Sep 16 13:31:21.465612 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.465595 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3-trusted-ca-bundle\") pod \"insights-operator-5844b856db-vdgqh\" (UID: \"647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3\") " pod="openshift-insights/insights-operator-5844b856db-vdgqh" Sep 16 13:31:21.466259 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.466240 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-hnlgr"] Sep 16 13:31:21.488091 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.488074 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-4xmxv"] Sep 16 13:31:21.488220 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.488204 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hnlgr" Sep 16 13:31:21.490962 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.490939 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 16 13:31:21.491057 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.491030 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 16 13:31:21.491285 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.491262 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:31:21.491285 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.491282 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 16 13:31:21.491424 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.491310 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-72lh2\"" Sep 16 13:31:21.508080 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.508058 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-4xmxv"] Sep 16 13:31:21.508155 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.508084 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-hnlgr"] Sep 16 13:31:21.508155 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.508152 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" Sep 16 13:31:21.511994 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.511972 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:31:21.512060 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.511994 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-42swj\"" Sep 16 13:31:21.512060 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.512044 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 16 13:31:21.512146 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.511996 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 16 13:31:21.512146 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.512097 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 16 13:31:21.517960 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.517944 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 16 13:31:21.566465 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.566441 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-stats-auth\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:21.566552 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.566478 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3-tmp\") pod \"insights-operator-5844b856db-vdgqh\" (UID: \"647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3\") " pod="openshift-insights/insights-operator-5844b856db-vdgqh" Sep 16 13:31:21.566552 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.566496 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j6l8\" (UniqueName: \"kubernetes.io/projected/6c074fd6-74bd-4ef3-a90f-a620fe95962c-kube-api-access-9j6l8\") pod \"cluster-samples-operator-86499c89c4-s2cmm\" (UID: \"6c074fd6-74bd-4ef3-a90f-a620fe95962c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s2cmm" Sep 16 13:31:21.566552 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.566511 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbc67ede-7ee2-4ac0-b256-952341815e01-service-ca-bundle\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:21.566663 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.566619 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-zb868\" (UID: \"1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-zb868" Sep 16 13:31:21.566663 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.566641 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lrj5\" (UniqueName: \"kubernetes.io/projected/1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b-kube-api-access-6lrj5\") pod \"kube-storage-version-migrator-operator-6c8f84f897-zb868\" (UID: \"1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-zb868" Sep 16 13:31:21.566754 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.566673 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcgz5\" (UniqueName: \"kubernetes.io/projected/7c11c223-afde-48fd-a3a2-e07f1cee8596-kube-api-access-rcgz5\") pod \"volume-data-source-validator-86695b7cc7-q995f\" (UID: \"7c11c223-afde-48fd-a3a2-e07f1cee8596\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q995f" Sep 16 13:31:21.566754 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.566703 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0d3da9f2-755a-47ea-b78c-2660c79e534c-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-pw5cd\" (UID: \"0d3da9f2-755a-47ea-b78c-2660c79e534c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pw5cd" Sep 16 13:31:21.566754 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.566720 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3-trusted-ca-bundle\") pod \"insights-operator-5844b856db-vdgqh\" (UID: \"647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3\") " pod="openshift-insights/insights-operator-5844b856db-vdgqh" Sep 16 13:31:21.566754 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.566734 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c074fd6-74bd-4ef3-a90f-a620fe95962c-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-s2cmm\" (UID: \"6c074fd6-74bd-4ef3-a90f-a620fe95962c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s2cmm" Sep 16 13:31:21.566754 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.566750 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-zb868\" (UID: \"1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-zb868" Sep 16 13:31:21.566933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.566768 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3-service-ca-bundle\") pod \"insights-operator-5844b856db-vdgqh\" (UID: \"647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3\") " pod="openshift-insights/insights-operator-5844b856db-vdgqh" Sep 16 13:31:21.566933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.566781 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3-serving-cert\") pod \"insights-operator-5844b856db-vdgqh\" (UID: \"647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3\") " pod="openshift-insights/insights-operator-5844b856db-vdgqh" Sep 16 13:31:21.566933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.566791 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3-tmp\") pod \"insights-operator-5844b856db-vdgqh\" (UID: \"647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3\") " pod="openshift-insights/insights-operator-5844b856db-vdgqh" Sep 16 13:31:21.566933 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.566823 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3-snapshots\") pod \"insights-operator-5844b856db-vdgqh\" (UID: \"647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3\") " pod="openshift-insights/insights-operator-5844b856db-vdgqh" Sep 16 13:31:21.567064 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.566969 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg49k\" (UniqueName: \"kubernetes.io/projected/fbc67ede-7ee2-4ac0-b256-952341815e01-kube-api-access-lg49k\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:21.567064 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.567002 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdbl7\" (UniqueName: \"kubernetes.io/projected/0d3da9f2-755a-47ea-b78c-2660c79e534c-kube-api-access-cdbl7\") pod \"cluster-monitoring-operator-6cd954b897-pw5cd\" (UID: \"0d3da9f2-755a-47ea-b78c-2660c79e534c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pw5cd" Sep 16 13:31:21.567064 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.567020 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-default-certificate\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:21.567176 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.567079 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl8jb\" (UniqueName: \"kubernetes.io/projected/647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3-kube-api-access-hl8jb\") pod \"insights-operator-5844b856db-vdgqh\" (UID: \"647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3\") " pod="openshift-insights/insights-operator-5844b856db-vdgqh" Sep 16 13:31:21.567176 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.567106 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d3da9f2-755a-47ea-b78c-2660c79e534c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-pw5cd\" (UID: \"0d3da9f2-755a-47ea-b78c-2660c79e534c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pw5cd" Sep 16 13:31:21.567176 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.567132 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-metrics-certs\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:21.567338 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.567318 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3-service-ca-bundle\") pod \"insights-operator-5844b856db-vdgqh\" (UID: \"647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3\") " pod="openshift-insights/insights-operator-5844b856db-vdgqh" Sep 16 13:31:21.567371 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.567334 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3-snapshots\") pod \"insights-operator-5844b856db-vdgqh\" (UID: \"647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3\") " pod="openshift-insights/insights-operator-5844b856db-vdgqh" Sep 16 13:31:21.567559 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.567543 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3-trusted-ca-bundle\") pod \"insights-operator-5844b856db-vdgqh\" (UID: \"647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3\") " pod="openshift-insights/insights-operator-5844b856db-vdgqh" Sep 16 13:31:21.569713 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.569693 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3-serving-cert\") pod \"insights-operator-5844b856db-vdgqh\" (UID: \"647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3\") " pod="openshift-insights/insights-operator-5844b856db-vdgqh" Sep 16 13:31:21.581523 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.581503 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d6bps" event={"ID":"870338a6-a8f2-46f8-a5e0-b1b17837057a","Type":"ContainerStarted","Data":"d38647a3f8d6bf1b2662f7371f0ed95004adad3a34a88fdf6843df5c9aa2d143"} Sep 16 13:31:21.584524 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.584503 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl8jb\" (UniqueName: \"kubernetes.io/projected/647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3-kube-api-access-hl8jb\") pod \"insights-operator-5844b856db-vdgqh\" (UID: \"647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3\") " pod="openshift-insights/insights-operator-5844b856db-vdgqh" Sep 16 13:31:21.593161 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.593141 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-vdgqh" Sep 16 13:31:21.608038 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.608005 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d6bps" podStartSLOduration=9.054116409 podStartE2EDuration="42.607993889s" totalImagesPullingTime="33.55387748s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:30:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:21.606858447 +0000 UTC m=+42.651161431" watchObservedRunningTime="2026-09-16 13:31:21.607993889 +0000 UTC m=+42.652296854" Sep 16 13:31:21.668534 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.667723 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdbl7\" (UniqueName: \"kubernetes.io/projected/0d3da9f2-755a-47ea-b78c-2660c79e534c-kube-api-access-cdbl7\") pod \"cluster-monitoring-operator-6cd954b897-pw5cd\" (UID: \"0d3da9f2-755a-47ea-b78c-2660c79e534c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pw5cd" Sep 16 13:31:21.668534 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.667767 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-default-certificate\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:21.668534 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.667798 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8618820e-6a4d-461e-bf89-c84a72b217fd-serving-cert\") pod \"service-ca-operator-cf7c776b7-hnlgr\" (UID: \"8618820e-6a4d-461e-bf89-c84a72b217fd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hnlgr" Sep 16 13:31:21.668534 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.667843 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7828efb-6be7-4f7b-986f-645a2370163a-serving-cert\") pod \"console-operator-696884fb9c-4xmxv\" (UID: \"c7828efb-6be7-4f7b-986f-645a2370163a\") " pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" Sep 16 13:31:21.668534 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.667900 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d3da9f2-755a-47ea-b78c-2660c79e534c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-pw5cd\" (UID: \"0d3da9f2-755a-47ea-b78c-2660c79e534c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pw5cd" Sep 16 13:31:21.668534 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.667928 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-metrics-certs\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:21.668534 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.667963 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-stats-auth\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:21.668534 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.668003 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j6l8\" (UniqueName: \"kubernetes.io/projected/6c074fd6-74bd-4ef3-a90f-a620fe95962c-kube-api-access-9j6l8\") pod \"cluster-samples-operator-86499c89c4-s2cmm\" (UID: \"6c074fd6-74bd-4ef3-a90f-a620fe95962c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s2cmm" Sep 16 13:31:21.668534 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.668028 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbc67ede-7ee2-4ac0-b256-952341815e01-service-ca-bundle\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:21.668534 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.668067 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-zb868\" (UID: \"1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-zb868" Sep 16 13:31:21.668534 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.668091 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lrj5\" (UniqueName: \"kubernetes.io/projected/1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b-kube-api-access-6lrj5\") pod \"kube-storage-version-migrator-operator-6c8f84f897-zb868\" (UID: \"1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-zb868" Sep 16 13:31:21.668534 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.668132 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcgz5\" (UniqueName: \"kubernetes.io/projected/7c11c223-afde-48fd-a3a2-e07f1cee8596-kube-api-access-rcgz5\") pod \"volume-data-source-validator-86695b7cc7-q995f\" (UID: \"7c11c223-afde-48fd-a3a2-e07f1cee8596\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q995f" Sep 16 13:31:21.668534 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.668161 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0d3da9f2-755a-47ea-b78c-2660c79e534c-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-pw5cd\" (UID: \"0d3da9f2-755a-47ea-b78c-2660c79e534c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pw5cd" Sep 16 13:31:21.668534 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.668187 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c074fd6-74bd-4ef3-a90f-a620fe95962c-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-s2cmm\" (UID: \"6c074fd6-74bd-4ef3-a90f-a620fe95962c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s2cmm" Sep 16 13:31:21.668534 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:21.668284 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 13:31:21.669576 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:21.668358 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbc67ede-7ee2-4ac0-b256-952341815e01-service-ca-bundle podName:fbc67ede-7ee2-4ac0-b256-952341815e01 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:22.168330544 +0000 UTC m=+43.212633775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fbc67ede-7ee2-4ac0-b256-952341815e01-service-ca-bundle") pod "router-default-858dfcf476-rkhcs" (UID: "fbc67ede-7ee2-4ac0-b256-952341815e01") : configmap references non-existent config key: service-ca.crt Sep 16 13:31:21.669576 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:21.668414 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3da9f2-755a-47ea-b78c-2660c79e534c-cluster-monitoring-operator-tls podName:0d3da9f2-755a-47ea-b78c-2660c79e534c nodeName:}" failed. No retries permitted until 2026-09-16 13:31:22.168403515 +0000 UTC m=+43.212706463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0d3da9f2-755a-47ea-b78c-2660c79e534c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-pw5cd" (UID: "0d3da9f2-755a-47ea-b78c-2660c79e534c") : secret "cluster-monitoring-operator-tls" not found Sep 16 13:31:21.669576 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.669182 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-zb868\" (UID: \"1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-zb868" Sep 16 13:31:21.669576 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.669212 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0d3da9f2-755a-47ea-b78c-2660c79e534c-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-pw5cd\" (UID: \"0d3da9f2-755a-47ea-b78c-2660c79e534c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pw5cd" Sep 16 13:31:21.669576 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.669242 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7828efb-6be7-4f7b-986f-645a2370163a-trusted-ca\") pod \"console-operator-696884fb9c-4xmxv\" (UID: \"c7828efb-6be7-4f7b-986f-645a2370163a\") " pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" Sep 16 13:31:21.669576 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.669278 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2c6m\" (UniqueName: \"kubernetes.io/projected/c7828efb-6be7-4f7b-986f-645a2370163a-kube-api-access-q2c6m\") pod \"console-operator-696884fb9c-4xmxv\" (UID: \"c7828efb-6be7-4f7b-986f-645a2370163a\") " pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" Sep 16 13:31:21.669576 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:21.669307 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 13:31:21.669576 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.669306 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8618820e-6a4d-461e-bf89-c84a72b217fd-config\") pod \"service-ca-operator-cf7c776b7-hnlgr\" (UID: \"8618820e-6a4d-461e-bf89-c84a72b217fd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hnlgr" Sep 16 13:31:21.669576 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:21.669360 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c074fd6-74bd-4ef3-a90f-a620fe95962c-samples-operator-tls podName:6c074fd6-74bd-4ef3-a90f-a620fe95962c nodeName:}" failed. No retries permitted until 2026-09-16 13:31:22.169346852 +0000 UTC m=+43.213649796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6c074fd6-74bd-4ef3-a90f-a620fe95962c-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-s2cmm" (UID: "6c074fd6-74bd-4ef3-a90f-a620fe95962c") : secret "samples-operator-tls" not found Sep 16 13:31:21.669576 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:21.669419 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 13:31:21.670055 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:21.669609 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-metrics-certs podName:fbc67ede-7ee2-4ac0-b256-952341815e01 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:22.169590456 +0000 UTC m=+43.213893415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-metrics-certs") pod "router-default-858dfcf476-rkhcs" (UID: "fbc67ede-7ee2-4ac0-b256-952341815e01") : secret "router-metrics-certs-default" not found Sep 16 13:31:21.670055 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.669639 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg49k\" (UniqueName: \"kubernetes.io/projected/fbc67ede-7ee2-4ac0-b256-952341815e01-kube-api-access-lg49k\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:21.670055 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.669673 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c7828efb-6be7-4f7b-986f-645a2370163a-config\") pod \"console-operator-696884fb9c-4xmxv\" (UID: \"c7828efb-6be7-4f7b-986f-645a2370163a\") " pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" Sep 16 13:31:21.670055 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.669698 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hgzz\" (UniqueName: \"kubernetes.io/projected/8618820e-6a4d-461e-bf89-c84a72b217fd-kube-api-access-8hgzz\") pod \"service-ca-operator-cf7c776b7-hnlgr\" (UID: \"8618820e-6a4d-461e-bf89-c84a72b217fd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hnlgr" Sep 16 13:31:21.670055 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.669810 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-zb868\" (UID: \"1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-zb868" Sep 16 13:31:21.672357 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.672333 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-zb868\" (UID: \"1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-zb868" Sep 16 13:31:21.672545 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.672527 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-default-certificate\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:21.672779 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.672760 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-stats-auth\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:21.678032 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.678017 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdbl7\" (UniqueName: \"kubernetes.io/projected/0d3da9f2-755a-47ea-b78c-2660c79e534c-kube-api-access-cdbl7\") pod \"cluster-monitoring-operator-6cd954b897-pw5cd\" (UID: \"0d3da9f2-755a-47ea-b78c-2660c79e534c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pw5cd" Sep 16 13:31:21.681753 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.681722 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j6l8\" (UniqueName: \"kubernetes.io/projected/6c074fd6-74bd-4ef3-a90f-a620fe95962c-kube-api-access-9j6l8\") pod \"cluster-samples-operator-86499c89c4-s2cmm\" (UID: \"6c074fd6-74bd-4ef3-a90f-a620fe95962c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s2cmm" Sep 16 13:31:21.685589 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.685544 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcgz5\" (UniqueName: \"kubernetes.io/projected/7c11c223-afde-48fd-a3a2-e07f1cee8596-kube-api-access-rcgz5\") pod \"volume-data-source-validator-86695b7cc7-q995f\" (UID: \"7c11c223-afde-48fd-a3a2-e07f1cee8596\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q995f" Sep 16 13:31:21.686237 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.686191 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lrj5\" (UniqueName: \"kubernetes.io/projected/1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b-kube-api-access-6lrj5\") pod \"kube-storage-version-migrator-operator-6c8f84f897-zb868\" (UID: \"1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-zb868" Sep 16 13:31:21.689231 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.689210 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg49k\" (UniqueName: \"kubernetes.io/projected/fbc67ede-7ee2-4ac0-b256-952341815e01-kube-api-access-lg49k\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:21.719840 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.719824 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q995f" Sep 16 13:31:21.729992 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.729973 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-zb868" Sep 16 13:31:21.770645 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.770624 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c7828efb-6be7-4f7b-986f-645a2370163a-config\") pod \"console-operator-696884fb9c-4xmxv\" (UID: \"c7828efb-6be7-4f7b-986f-645a2370163a\") " pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" Sep 16 13:31:21.770726 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.770657 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hgzz\" (UniqueName: \"kubernetes.io/projected/8618820e-6a4d-461e-bf89-c84a72b217fd-kube-api-access-8hgzz\") pod \"service-ca-operator-cf7c776b7-hnlgr\" (UID: \"8618820e-6a4d-461e-bf89-c84a72b217fd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hnlgr" Sep 16 13:31:21.770726 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.770712 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8618820e-6a4d-461e-bf89-c84a72b217fd-serving-cert\") pod \"service-ca-operator-cf7c776b7-hnlgr\" (UID: \"8618820e-6a4d-461e-bf89-c84a72b217fd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hnlgr" Sep 16 13:31:21.770796 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.770738 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7828efb-6be7-4f7b-986f-645a2370163a-serving-cert\") pod \"console-operator-696884fb9c-4xmxv\" (UID: \"c7828efb-6be7-4f7b-986f-645a2370163a\") " pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" Sep 16 13:31:21.770940 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.770925 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7828efb-6be7-4f7b-986f-645a2370163a-trusted-ca\") pod \"console-operator-696884fb9c-4xmxv\" (UID: \"c7828efb-6be7-4f7b-986f-645a2370163a\") " pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" Sep 16 13:31:21.770976 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.770956 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2c6m\" (UniqueName: \"kubernetes.io/projected/c7828efb-6be7-4f7b-986f-645a2370163a-kube-api-access-q2c6m\") pod \"console-operator-696884fb9c-4xmxv\" (UID: \"c7828efb-6be7-4f7b-986f-645a2370163a\") " pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" Sep 16 13:31:21.771004 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.770983 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8618820e-6a4d-461e-bf89-c84a72b217fd-config\") pod \"service-ca-operator-cf7c776b7-hnlgr\" (UID: \"8618820e-6a4d-461e-bf89-c84a72b217fd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hnlgr" Sep 16 13:31:21.771771 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.771593 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8618820e-6a4d-461e-bf89-c84a72b217fd-config\") pod \"service-ca-operator-cf7c776b7-hnlgr\" (UID: \"8618820e-6a4d-461e-bf89-c84a72b217fd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hnlgr" Sep 16 13:31:21.771771 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.771622 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c7828efb-6be7-4f7b-986f-645a2370163a-config\") pod \"console-operator-696884fb9c-4xmxv\" (UID: \"c7828efb-6be7-4f7b-986f-645a2370163a\") " pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" Sep 16 13:31:21.772535 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.772510 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7828efb-6be7-4f7b-986f-645a2370163a-trusted-ca\") pod \"console-operator-696884fb9c-4xmxv\" (UID: \"c7828efb-6be7-4f7b-986f-645a2370163a\") " pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" Sep 16 13:31:21.773408 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.773358 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8618820e-6a4d-461e-bf89-c84a72b217fd-serving-cert\") pod \"service-ca-operator-cf7c776b7-hnlgr\" (UID: \"8618820e-6a4d-461e-bf89-c84a72b217fd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hnlgr" Sep 16 13:31:21.773912 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.773892 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7828efb-6be7-4f7b-986f-645a2370163a-serving-cert\") pod \"console-operator-696884fb9c-4xmxv\" (UID: \"c7828efb-6be7-4f7b-986f-645a2370163a\") " pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" Sep 16 13:31:21.779881 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.779855 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2c6m\" (UniqueName: \"kubernetes.io/projected/c7828efb-6be7-4f7b-986f-645a2370163a-kube-api-access-q2c6m\") pod \"console-operator-696884fb9c-4xmxv\" (UID: \"c7828efb-6be7-4f7b-986f-645a2370163a\") " pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" Sep 16 13:31:21.779957 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.779910 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hgzz\" (UniqueName: \"kubernetes.io/projected/8618820e-6a4d-461e-bf89-c84a72b217fd-kube-api-access-8hgzz\") pod \"service-ca-operator-cf7c776b7-hnlgr\" (UID: \"8618820e-6a4d-461e-bf89-c84a72b217fd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hnlgr" Sep 16 13:31:21.797029 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.796791 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hnlgr" Sep 16 13:31:21.816195 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.816124 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" Sep 16 13:31:21.889941 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:21.889918 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-vdgqh"] Sep 16 13:31:22.010582 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:22.010523 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q995f"] Sep 16 13:31:22.012314 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:31:22.012292 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c11c223_afde_48fd_a3a2_e07f1cee8596.slice/crio-8f62d53efbfd76981f0c4cd4fb39a86df5ab1c461e382ccd065779cc6ee2c0b6 WatchSource:0}: Error finding container 8f62d53efbfd76981f0c4cd4fb39a86df5ab1c461e382ccd065779cc6ee2c0b6: Status 404 returned error can't find the container with id 8f62d53efbfd76981f0c4cd4fb39a86df5ab1c461e382ccd065779cc6ee2c0b6 Sep 16 13:31:22.047626 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:22.047606 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-zb868"] Sep 16 13:31:22.049758 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:31:22.049736 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e0d4cff_caad_4c86_bbd8_e0b1e86e3d7b.slice/crio-0d3d12fe65cb44c045aece9b7e8a4325aa31709135702584063f1e023d5d3137 WatchSource:0}: Error finding container 0d3d12fe65cb44c045aece9b7e8a4325aa31709135702584063f1e023d5d3137: Status 404 returned error can't find the container with id 0d3d12fe65cb44c045aece9b7e8a4325aa31709135702584063f1e023d5d3137 Sep 16 13:31:22.099438 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:22.099347 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-hnlgr"] Sep 16 13:31:22.101501 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:31:22.101482 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8618820e_6a4d_461e_bf89_c84a72b217fd.slice/crio-e509c98dedb18b907cdc05f53335a7e9de83e04941c4a7d8018d3be4aaa5d106 WatchSource:0}: Error finding container e509c98dedb18b907cdc05f53335a7e9de83e04941c4a7d8018d3be4aaa5d106: Status 404 returned error can't find the container with id e509c98dedb18b907cdc05f53335a7e9de83e04941c4a7d8018d3be4aaa5d106 Sep 16 13:31:22.123064 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:22.122983 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-4xmxv"] Sep 16 13:31:22.125275 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:31:22.125254 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7828efb_6be7_4f7b_986f_645a2370163a.slice/crio-b5a2b074f79c2306f9d1001a6ada0cf05174cafed547235378a130d3494ce619 WatchSource:0}: Error finding container b5a2b074f79c2306f9d1001a6ada0cf05174cafed547235378a130d3494ce619: Status 404 returned error can't find the container with id b5a2b074f79c2306f9d1001a6ada0cf05174cafed547235378a130d3494ce619 Sep 16 13:31:22.175812 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:22.175794 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d3da9f2-755a-47ea-b78c-2660c79e534c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-pw5cd\" (UID: \"0d3da9f2-755a-47ea-b78c-2660c79e534c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pw5cd" Sep 16 13:31:22.175914 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:22.175822 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-metrics-certs\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:22.175914 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:22.175851 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbc67ede-7ee2-4ac0-b256-952341815e01-service-ca-bundle\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:22.175914 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:22.175887 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c074fd6-74bd-4ef3-a90f-a620fe95962c-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-s2cmm\" (UID: \"6c074fd6-74bd-4ef3-a90f-a620fe95962c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s2cmm" Sep 16 13:31:22.176042 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:22.175924 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 13:31:22.176042 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:22.175950 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 13:31:22.176042 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:22.175963 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 13:31:22.176042 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:22.175978 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3da9f2-755a-47ea-b78c-2660c79e534c-cluster-monitoring-operator-tls podName:0d3da9f2-755a-47ea-b78c-2660c79e534c nodeName:}" failed. No retries permitted until 2026-09-16 13:31:23.175958309 +0000 UTC m=+44.220261253 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0d3da9f2-755a-47ea-b78c-2660c79e534c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-pw5cd" (UID: "0d3da9f2-755a-47ea-b78c-2660c79e534c") : secret "cluster-monitoring-operator-tls" not found Sep 16 13:31:22.176042 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:22.175996 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c074fd6-74bd-4ef3-a90f-a620fe95962c-samples-operator-tls podName:6c074fd6-74bd-4ef3-a90f-a620fe95962c nodeName:}" failed. No retries permitted until 2026-09-16 13:31:23.175985905 +0000 UTC m=+44.220288849 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6c074fd6-74bd-4ef3-a90f-a620fe95962c-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-s2cmm" (UID: "6c074fd6-74bd-4ef3-a90f-a620fe95962c") : secret "samples-operator-tls" not found Sep 16 13:31:22.176042 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:22.176008 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-metrics-certs podName:fbc67ede-7ee2-4ac0-b256-952341815e01 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:23.17600236 +0000 UTC m=+44.220305304 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-metrics-certs") pod "router-default-858dfcf476-rkhcs" (UID: "fbc67ede-7ee2-4ac0-b256-952341815e01") : secret "router-metrics-certs-default" not found Sep 16 13:31:22.176042 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:22.176017 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbc67ede-7ee2-4ac0-b256-952341815e01-service-ca-bundle podName:fbc67ede-7ee2-4ac0-b256-952341815e01 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:23.176012458 +0000 UTC m=+44.220315403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fbc67ede-7ee2-4ac0-b256-952341815e01-service-ca-bundle") pod "router-default-858dfcf476-rkhcs" (UID: "fbc67ede-7ee2-4ac0-b256-952341815e01") : configmap references non-existent config key: service-ca.crt Sep 16 13:31:22.585449 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:22.585415 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" event={"ID":"c7828efb-6be7-4f7b-986f-645a2370163a","Type":"ContainerStarted","Data":"b5a2b074f79c2306f9d1001a6ada0cf05174cafed547235378a130d3494ce619"} Sep 16 13:31:22.586462 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:22.586437 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q995f" event={"ID":"7c11c223-afde-48fd-a3a2-e07f1cee8596","Type":"ContainerStarted","Data":"8f62d53efbfd76981f0c4cd4fb39a86df5ab1c461e382ccd065779cc6ee2c0b6"} Sep 16 13:31:22.587424 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:22.587398 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-zb868" event={"ID":"1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b","Type":"ContainerStarted","Data":"0d3d12fe65cb44c045aece9b7e8a4325aa31709135702584063f1e023d5d3137"} Sep 16 13:31:22.588386 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:22.588352 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-vdgqh" event={"ID":"647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3","Type":"ContainerStarted","Data":"4c399a4ff4ee30aaaff1c1bdcdb4733128d2376c8d285df625812d699c1d91ca"} Sep 16 13:31:22.589459 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:22.589442 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hnlgr" event={"ID":"8618820e-6a4d-461e-bf89-c84a72b217fd","Type":"ContainerStarted","Data":"e509c98dedb18b907cdc05f53335a7e9de83e04941c4a7d8018d3be4aaa5d106"} Sep 16 13:31:23.186674 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:23.185574 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c074fd6-74bd-4ef3-a90f-a620fe95962c-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-s2cmm\" (UID: \"6c074fd6-74bd-4ef3-a90f-a620fe95962c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s2cmm" Sep 16 13:31:23.186674 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:23.185696 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d3da9f2-755a-47ea-b78c-2660c79e534c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-pw5cd\" (UID: \"0d3da9f2-755a-47ea-b78c-2660c79e534c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pw5cd" Sep 16 13:31:23.186674 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:23.185726 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-metrics-certs\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:23.186674 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:23.185769 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbc67ede-7ee2-4ac0-b256-952341815e01-service-ca-bundle\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:23.186674 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:23.185945 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbc67ede-7ee2-4ac0-b256-952341815e01-service-ca-bundle podName:fbc67ede-7ee2-4ac0-b256-952341815e01 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:25.185917436 +0000 UTC m=+46.230220396 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fbc67ede-7ee2-4ac0-b256-952341815e01-service-ca-bundle") pod "router-default-858dfcf476-rkhcs" (UID: "fbc67ede-7ee2-4ac0-b256-952341815e01") : configmap references non-existent config key: service-ca.crt Sep 16 13:31:23.186674 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:23.186345 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 13:31:23.186674 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:23.186414 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c074fd6-74bd-4ef3-a90f-a620fe95962c-samples-operator-tls podName:6c074fd6-74bd-4ef3-a90f-a620fe95962c nodeName:}" failed. No retries permitted until 2026-09-16 13:31:25.186396584 +0000 UTC m=+46.230699528 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6c074fd6-74bd-4ef3-a90f-a620fe95962c-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-s2cmm" (UID: "6c074fd6-74bd-4ef3-a90f-a620fe95962c") : secret "samples-operator-tls" not found Sep 16 13:31:23.186674 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:23.186490 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 13:31:23.186674 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:23.186526 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3da9f2-755a-47ea-b78c-2660c79e534c-cluster-monitoring-operator-tls podName:0d3da9f2-755a-47ea-b78c-2660c79e534c nodeName:}" failed. No retries permitted until 2026-09-16 13:31:25.186515389 +0000 UTC m=+46.230818347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0d3da9f2-755a-47ea-b78c-2660c79e534c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-pw5cd" (UID: "0d3da9f2-755a-47ea-b78c-2660c79e534c") : secret "cluster-monitoring-operator-tls" not found Sep 16 13:31:23.186674 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:23.186573 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 13:31:23.186674 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:23.186598 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-metrics-certs podName:fbc67ede-7ee2-4ac0-b256-952341815e01 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:25.186588663 +0000 UTC m=+46.230891607 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-metrics-certs") pod "router-default-858dfcf476-rkhcs" (UID: "fbc67ede-7ee2-4ac0-b256-952341815e01") : secret "router-metrics-certs-default" not found Sep 16 13:31:25.203827 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:25.203791 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d3da9f2-755a-47ea-b78c-2660c79e534c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-pw5cd\" (UID: \"0d3da9f2-755a-47ea-b78c-2660c79e534c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pw5cd" Sep 16 13:31:25.203827 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:25.203832 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-metrics-certs\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:25.204445 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:25.203943 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 13:31:25.204445 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:25.203948 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 13:31:25.204445 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:25.203994 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbc67ede-7ee2-4ac0-b256-952341815e01-service-ca-bundle\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:25.204445 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:25.204004 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-metrics-certs podName:fbc67ede-7ee2-4ac0-b256-952341815e01 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:29.203985884 +0000 UTC m=+50.248288828 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-metrics-certs") pod "router-default-858dfcf476-rkhcs" (UID: "fbc67ede-7ee2-4ac0-b256-952341815e01") : secret "router-metrics-certs-default" not found Sep 16 13:31:25.204445 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:25.204039 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3da9f2-755a-47ea-b78c-2660c79e534c-cluster-monitoring-operator-tls podName:0d3da9f2-755a-47ea-b78c-2660c79e534c nodeName:}" failed. No retries permitted until 2026-09-16 13:31:29.20402242 +0000 UTC m=+50.248325364 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0d3da9f2-755a-47ea-b78c-2660c79e534c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-pw5cd" (UID: "0d3da9f2-755a-47ea-b78c-2660c79e534c") : secret "cluster-monitoring-operator-tls" not found Sep 16 13:31:25.204445 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:25.204083 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbc67ede-7ee2-4ac0-b256-952341815e01-service-ca-bundle podName:fbc67ede-7ee2-4ac0-b256-952341815e01 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:29.20406935 +0000 UTC m=+50.248372299 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fbc67ede-7ee2-4ac0-b256-952341815e01-service-ca-bundle") pod "router-default-858dfcf476-rkhcs" (UID: "fbc67ede-7ee2-4ac0-b256-952341815e01") : configmap references non-existent config key: service-ca.crt Sep 16 13:31:25.204445 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:25.204153 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c074fd6-74bd-4ef3-a90f-a620fe95962c-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-s2cmm\" (UID: \"6c074fd6-74bd-4ef3-a90f-a620fe95962c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s2cmm" Sep 16 13:31:25.204445 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:25.204290 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 13:31:25.204445 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:25.204345 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c074fd6-74bd-4ef3-a90f-a620fe95962c-samples-operator-tls podName:6c074fd6-74bd-4ef3-a90f-a620fe95962c nodeName:}" failed. No retries permitted until 2026-09-16 13:31:29.204331992 +0000 UTC m=+50.248634956 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6c074fd6-74bd-4ef3-a90f-a620fe95962c-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-s2cmm" (UID: "6c074fd6-74bd-4ef3-a90f-a620fe95962c") : secret "samples-operator-tls" not found Sep 16 13:31:26.163869 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:26.163824 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-6hxrk"] Sep 16 13:31:26.197432 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:26.197403 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-6hxrk"] Sep 16 13:31:26.197572 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:26.197543 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-6hxrk" Sep 16 13:31:26.200481 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:26.200457 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-45rqq\"" Sep 16 13:31:26.211163 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:26.211141 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntflc\" (UniqueName: \"kubernetes.io/projected/96770fb2-49ff-487d-831c-4579e2b91dc4-kube-api-access-ntflc\") pod \"network-check-source-6f8b768455-6hxrk\" (UID: \"96770fb2-49ff-487d-831c-4579e2b91dc4\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-6hxrk" Sep 16 13:31:26.312339 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:26.312308 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntflc\" (UniqueName: \"kubernetes.io/projected/96770fb2-49ff-487d-831c-4579e2b91dc4-kube-api-access-ntflc\") pod \"network-check-source-6f8b768455-6hxrk\" (UID: \"96770fb2-49ff-487d-831c-4579e2b91dc4\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-6hxrk" Sep 16 13:31:26.321618 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:26.321590 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntflc\" (UniqueName: \"kubernetes.io/projected/96770fb2-49ff-487d-831c-4579e2b91dc4-kube-api-access-ntflc\") pod \"network-check-source-6f8b768455-6hxrk\" (UID: \"96770fb2-49ff-487d-831c-4579e2b91dc4\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-6hxrk" Sep 16 13:31:26.508959 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:26.508880 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-6hxrk" Sep 16 13:31:27.093716 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:27.093695 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-6hxrk"] Sep 16 13:31:27.119229 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:31:27.119203 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96770fb2_49ff_487d_831c_4579e2b91dc4.slice/crio-e508b82e8e0b1e7bae139c5c257aa750effaef3f56734c5e184265e035f6228b WatchSource:0}: Error finding container e508b82e8e0b1e7bae139c5c257aa750effaef3f56734c5e184265e035f6228b: Status 404 returned error can't find the container with id e508b82e8e0b1e7bae139c5c257aa750effaef3f56734c5e184265e035f6228b Sep 16 13:31:27.609834 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:27.609791 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-zb868" event={"ID":"1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b","Type":"ContainerStarted","Data":"c388f5c713f8187c9a42ce29e84759ace04823829b44f74b1d8e77d2ec200a12"} Sep 16 13:31:27.611550 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:27.611521 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-vdgqh" event={"ID":"647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3","Type":"ContainerStarted","Data":"0bf800124a0716c7dc41382bd93c24610b893571029483f307096b3bb844b3e1"} Sep 16 13:31:27.612867 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:27.612842 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-6hxrk" event={"ID":"96770fb2-49ff-487d-831c-4579e2b91dc4","Type":"ContainerStarted","Data":"e508b82e8e0b1e7bae139c5c257aa750effaef3f56734c5e184265e035f6228b"} Sep 16 13:31:27.614415 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:27.614371 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hnlgr" event={"ID":"8618820e-6a4d-461e-bf89-c84a72b217fd","Type":"ContainerStarted","Data":"06d5a50a051b7e6f420feebe102ef7c6011b590031f36b5f6681a4b6f479bd83"} Sep 16 13:31:27.616143 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:27.616123 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4xmxv_c7828efb-6be7-4f7b-986f-645a2370163a/console-operator/0.log" Sep 16 13:31:27.616242 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:27.616157 2562 generic.go:356] "Generic (PLEG): container finished" podID="c7828efb-6be7-4f7b-986f-645a2370163a" containerID="1e08a87b44e518f44c1173db9adc06dc975ad4e4fffbadc5b2350277809963a4" exitCode=255 Sep 16 13:31:27.616398 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:27.616352 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" event={"ID":"c7828efb-6be7-4f7b-986f-645a2370163a","Type":"ContainerDied","Data":"1e08a87b44e518f44c1173db9adc06dc975ad4e4fffbadc5b2350277809963a4"} Sep 16 13:31:27.616502 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:27.616452 2562 scope.go:117] "RemoveContainer" containerID="1e08a87b44e518f44c1173db9adc06dc975ad4e4fffbadc5b2350277809963a4" Sep 16 13:31:27.618347 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:27.618325 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q995f" event={"ID":"7c11c223-afde-48fd-a3a2-e07f1cee8596","Type":"ContainerStarted","Data":"ef599fbdf2b122abd6fe4794e71dd90e3ffa6e76929f8d135c731c96ed58290d"} Sep 16 13:31:27.637335 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:27.637301 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-zb868" podStartSLOduration=1.855880357 podStartE2EDuration="6.637288557s" totalImagesPullingTime="4.7814082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:27.636164939 +0000 UTC m=+48.680467923" watchObservedRunningTime="2026-09-16 13:31:27.637288557 +0000 UTC m=+48.681591521" Sep 16 13:31:27.659008 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:27.658969 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hnlgr" podStartSLOduration=1.93227108 podStartE2EDuration="6.658956649s" totalImagesPullingTime="4.726685569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:27.65850553 +0000 UTC m=+48.702808498" watchObservedRunningTime="2026-09-16 13:31:27.658956649 +0000 UTC m=+48.703259611" Sep 16 13:31:27.696579 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:27.696355 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-vdgqh" podStartSLOduration=1.7623484729999999 podStartE2EDuration="6.696337314s" totalImagesPullingTime="4.933988841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:27.696119007 +0000 UTC m=+48.740422018" watchObservedRunningTime="2026-09-16 13:31:27.696337314 +0000 UTC m=+48.740640277" Sep 16 13:31:27.716307 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:27.716258 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q995f" podStartSLOduration=4.988561386 podStartE2EDuration="6.716239478s" totalImagesPullingTime="1.727678092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:27.715516724 +0000 UTC m=+48.759819705" watchObservedRunningTime="2026-09-16 13:31:27.716239478 +0000 UTC m=+48.760542444" Sep 16 13:31:28.011779 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.011745 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hvzrx"] Sep 16 13:31:28.030228 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.030199 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hvzrx"] Sep 16 13:31:28.030402 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.030314 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:28.033778 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.033732 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 16 13:31:28.034233 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.034211 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 16 13:31:28.034233 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.034223 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7rd5w\"" Sep 16 13:31:28.129026 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.128991 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f940b017-c539-41e8-be1a-c61f0f9ae25d-crio-socket\") pod \"insights-runtime-extractor-hvzrx\" (UID: \"f940b017-c539-41e8-be1a-c61f0f9ae25d\") " pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:28.129026 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.129051 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f940b017-c539-41e8-be1a-c61f0f9ae25d-data-volume\") pod \"insights-runtime-extractor-hvzrx\" (UID: \"f940b017-c539-41e8-be1a-c61f0f9ae25d\") " pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:28.129026 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.129178 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f940b017-c539-41e8-be1a-c61f0f9ae25d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvzrx\" (UID: \"f940b017-c539-41e8-be1a-c61f0f9ae25d\") " pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:28.129507 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.129311 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f940b017-c539-41e8-be1a-c61f0f9ae25d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvzrx\" (UID: \"f940b017-c539-41e8-be1a-c61f0f9ae25d\") " pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:28.129507 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.129352 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkdmp\" (UniqueName: \"kubernetes.io/projected/f940b017-c539-41e8-be1a-c61f0f9ae25d-kube-api-access-bkdmp\") pod \"insights-runtime-extractor-hvzrx\" (UID: \"f940b017-c539-41e8-be1a-c61f0f9ae25d\") " pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:28.230259 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.230208 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f940b017-c539-41e8-be1a-c61f0f9ae25d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvzrx\" (UID: \"f940b017-c539-41e8-be1a-c61f0f9ae25d\") " pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:28.230460 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.230343 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f940b017-c539-41e8-be1a-c61f0f9ae25d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvzrx\" (UID: \"f940b017-c539-41e8-be1a-c61f0f9ae25d\") " pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:28.230460 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:28.230355 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 13:31:28.230570 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.230455 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkdmp\" (UniqueName: \"kubernetes.io/projected/f940b017-c539-41e8-be1a-c61f0f9ae25d-kube-api-access-bkdmp\") pod \"insights-runtime-extractor-hvzrx\" (UID: \"f940b017-c539-41e8-be1a-c61f0f9ae25d\") " pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:28.230570 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:28.230460 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:31:28.230570 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:28.230488 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f940b017-c539-41e8-be1a-c61f0f9ae25d-kube-rbac-proxy-cm podName:f940b017-c539-41e8-be1a-c61f0f9ae25d nodeName:}" failed. No retries permitted until 2026-09-16 13:31:28.730469372 +0000 UTC m=+49.774772316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f940b017-c539-41e8-be1a-c61f0f9ae25d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hvzrx" (UID: "f940b017-c539-41e8-be1a-c61f0f9ae25d") : configmap "kube-rbac-proxy" not found Sep 16 13:31:28.230570 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:28.230538 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f940b017-c539-41e8-be1a-c61f0f9ae25d-insights-runtime-extractor-tls podName:f940b017-c539-41e8-be1a-c61f0f9ae25d nodeName:}" failed. No retries permitted until 2026-09-16 13:31:28.73052295 +0000 UTC m=+49.774825914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f940b017-c539-41e8-be1a-c61f0f9ae25d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvzrx" (UID: "f940b017-c539-41e8-be1a-c61f0f9ae25d") : secret "insights-runtime-extractor-tls" not found Sep 16 13:31:28.230570 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.230568 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f940b017-c539-41e8-be1a-c61f0f9ae25d-crio-socket\") pod \"insights-runtime-extractor-hvzrx\" (UID: \"f940b017-c539-41e8-be1a-c61f0f9ae25d\") " pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:28.230748 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.230594 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f940b017-c539-41e8-be1a-c61f0f9ae25d-data-volume\") pod \"insights-runtime-extractor-hvzrx\" (UID: \"f940b017-c539-41e8-be1a-c61f0f9ae25d\") " pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:28.230786 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.230751 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f940b017-c539-41e8-be1a-c61f0f9ae25d-crio-socket\") pod \"insights-runtime-extractor-hvzrx\" (UID: \"f940b017-c539-41e8-be1a-c61f0f9ae25d\") " pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:28.230897 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.230878 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f940b017-c539-41e8-be1a-c61f0f9ae25d-data-volume\") pod \"insights-runtime-extractor-hvzrx\" (UID: \"f940b017-c539-41e8-be1a-c61f0f9ae25d\") " pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:28.241483 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.241456 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkdmp\" (UniqueName: \"kubernetes.io/projected/f940b017-c539-41e8-be1a-c61f0f9ae25d-kube-api-access-bkdmp\") pod \"insights-runtime-extractor-hvzrx\" (UID: \"f940b017-c539-41e8-be1a-c61f0f9ae25d\") " pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:28.624005 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.623965 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4xmxv_c7828efb-6be7-4f7b-986f-645a2370163a/console-operator/1.log" Sep 16 13:31:28.624486 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.624468 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4xmxv_c7828efb-6be7-4f7b-986f-645a2370163a/console-operator/0.log" Sep 16 13:31:28.624556 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.624512 2562 generic.go:356] "Generic (PLEG): container finished" podID="c7828efb-6be7-4f7b-986f-645a2370163a" containerID="fc4b744a9e8d661e945c7f5d1c13ed1437f2363d75c049ce19803fd501cd2381" exitCode=255 Sep 16 13:31:28.624623 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.624605 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" event={"ID":"c7828efb-6be7-4f7b-986f-645a2370163a","Type":"ContainerDied","Data":"fc4b744a9e8d661e945c7f5d1c13ed1437f2363d75c049ce19803fd501cd2381"} Sep 16 13:31:28.624676 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.624662 2562 scope.go:117] "RemoveContainer" containerID="1e08a87b44e518f44c1173db9adc06dc975ad4e4fffbadc5b2350277809963a4" Sep 16 13:31:28.625324 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.625306 2562 scope.go:117] "RemoveContainer" containerID="fc4b744a9e8d661e945c7f5d1c13ed1437f2363d75c049ce19803fd501cd2381" Sep 16 13:31:28.625674 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:28.625455 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-4xmxv_openshift-console-operator(c7828efb-6be7-4f7b-986f-645a2370163a)\"" pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" podUID="c7828efb-6be7-4f7b-986f-645a2370163a" Sep 16 13:31:28.736010 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.735976 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f940b017-c539-41e8-be1a-c61f0f9ae25d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvzrx\" (UID: \"f940b017-c539-41e8-be1a-c61f0f9ae25d\") " pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:28.736118 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:28.736085 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 13:31:28.736166 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:28.736132 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f940b017-c539-41e8-be1a-c61f0f9ae25d-kube-rbac-proxy-cm podName:f940b017-c539-41e8-be1a-c61f0f9ae25d nodeName:}" failed. No retries permitted until 2026-09-16 13:31:29.73611744 +0000 UTC m=+50.780420385 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f940b017-c539-41e8-be1a-c61f0f9ae25d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hvzrx" (UID: "f940b017-c539-41e8-be1a-c61f0f9ae25d") : configmap "kube-rbac-proxy" not found Sep 16 13:31:28.736208 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:28.736164 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f940b017-c539-41e8-be1a-c61f0f9ae25d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvzrx\" (UID: \"f940b017-c539-41e8-be1a-c61f0f9ae25d\") " pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:28.736247 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:28.736238 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:31:28.736288 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:28.736275 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f940b017-c539-41e8-be1a-c61f0f9ae25d-insights-runtime-extractor-tls podName:f940b017-c539-41e8-be1a-c61f0f9ae25d nodeName:}" failed. No retries permitted until 2026-09-16 13:31:29.736264505 +0000 UTC m=+50.780567449 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f940b017-c539-41e8-be1a-c61f0f9ae25d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvzrx" (UID: "f940b017-c539-41e8-be1a-c61f0f9ae25d") : secret "insights-runtime-extractor-tls" not found Sep 16 13:31:29.240473 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:29.240440 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c074fd6-74bd-4ef3-a90f-a620fe95962c-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-s2cmm\" (UID: \"6c074fd6-74bd-4ef3-a90f-a620fe95962c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s2cmm" Sep 16 13:31:29.240649 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:29.240534 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d3da9f2-755a-47ea-b78c-2660c79e534c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-pw5cd\" (UID: \"0d3da9f2-755a-47ea-b78c-2660c79e534c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pw5cd" Sep 16 13:31:29.240649 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:29.240556 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-metrics-certs\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:29.240649 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:29.240576 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/390ee01e-b850-45d2-b21d-c6b002a1c0ff-metrics-tls\") pod \"dns-default-mxnjp\" (UID: \"390ee01e-b850-45d2-b21d-c6b002a1c0ff\") " pod="openshift-dns/dns-default-mxnjp" Sep 16 13:31:29.240649 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:29.240595 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbc67ede-7ee2-4ac0-b256-952341815e01-service-ca-bundle\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:29.240649 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:29.240592 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 13:31:29.240649 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:29.240631 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 13:31:29.240926 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:29.240662 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c074fd6-74bd-4ef3-a90f-a620fe95962c-samples-operator-tls podName:6c074fd6-74bd-4ef3-a90f-a620fe95962c nodeName:}" failed. No retries permitted until 2026-09-16 13:31:37.240643418 +0000 UTC m=+58.284946378 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6c074fd6-74bd-4ef3-a90f-a620fe95962c-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-s2cmm" (UID: "6c074fd6-74bd-4ef3-a90f-a620fe95962c") : secret "samples-operator-tls" not found Sep 16 13:31:29.240926 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:29.240696 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-tls\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:29.240926 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:29.240712 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3da9f2-755a-47ea-b78c-2660c79e534c-cluster-monitoring-operator-tls podName:0d3da9f2-755a-47ea-b78c-2660c79e534c nodeName:}" failed. No retries permitted until 2026-09-16 13:31:37.240696213 +0000 UTC m=+58.284999157 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0d3da9f2-755a-47ea-b78c-2660c79e534c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-pw5cd" (UID: "0d3da9f2-755a-47ea-b78c-2660c79e534c") : secret "cluster-monitoring-operator-tls" not found Sep 16 13:31:29.240926 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:29.240728 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 13:31:29.240926 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:29.240742 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 13:31:29.240926 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:29.240758 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 13:31:29.240926 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:29.240735 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbc67ede-7ee2-4ac0-b256-952341815e01-service-ca-bundle podName:fbc67ede-7ee2-4ac0-b256-952341815e01 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:37.240726137 +0000 UTC m=+58.285029081 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fbc67ede-7ee2-4ac0-b256-952341815e01-service-ca-bundle") pod "router-default-858dfcf476-rkhcs" (UID: "fbc67ede-7ee2-4ac0-b256-952341815e01") : configmap references non-existent config key: service-ca.crt Sep 16 13:31:29.240926 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:29.240769 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6449dbdc57-jc64r: secret "image-registry-tls" not found Sep 16 13:31:29.240926 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:29.240795 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/390ee01e-b850-45d2-b21d-c6b002a1c0ff-metrics-tls podName:390ee01e-b850-45d2-b21d-c6b002a1c0ff nodeName:}" failed. No retries permitted until 2026-09-16 13:31:45.240780543 +0000 UTC m=+66.285083492 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/390ee01e-b850-45d2-b21d-c6b002a1c0ff-metrics-tls") pod "dns-default-mxnjp" (UID: "390ee01e-b850-45d2-b21d-c6b002a1c0ff") : secret "dns-default-metrics-tls" not found Sep 16 13:31:29.240926 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:29.240813 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-metrics-certs podName:fbc67ede-7ee2-4ac0-b256-952341815e01 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:37.240804296 +0000 UTC m=+58.285107244 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-metrics-certs") pod "router-default-858dfcf476-rkhcs" (UID: "fbc67ede-7ee2-4ac0-b256-952341815e01") : secret "router-metrics-certs-default" not found Sep 16 13:31:29.240926 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:29.240838 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-tls podName:58c0f9ae-bb5d-4621-b804-29b5a3a88f6c nodeName:}" failed. No retries permitted until 2026-09-16 13:31:45.240829067 +0000 UTC m=+66.285132012 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-tls") pod "image-registry-6449dbdc57-jc64r" (UID: "58c0f9ae-bb5d-4621-b804-29b5a3a88f6c") : secret "image-registry-tls" not found Sep 16 13:31:29.630237 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:29.630168 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4xmxv_c7828efb-6be7-4f7b-986f-645a2370163a/console-operator/1.log" Sep 16 13:31:29.630638 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:29.630560 2562 scope.go:117] "RemoveContainer" containerID="fc4b744a9e8d661e945c7f5d1c13ed1437f2363d75c049ce19803fd501cd2381" Sep 16 13:31:29.630760 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:29.630736 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-4xmxv_openshift-console-operator(c7828efb-6be7-4f7b-986f-645a2370163a)\"" pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" podUID="c7828efb-6be7-4f7b-986f-645a2370163a" Sep 16 13:31:29.746021 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:29.745981 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f940b017-c539-41e8-be1a-c61f0f9ae25d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvzrx\" (UID: \"f940b017-c539-41e8-be1a-c61f0f9ae25d\") " pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:29.746214 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:29.746144 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f940b017-c539-41e8-be1a-c61f0f9ae25d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvzrx\" (UID: \"f940b017-c539-41e8-be1a-c61f0f9ae25d\") " pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:29.746280 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:29.746260 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 13:31:29.746280 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:29.746273 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:31:29.746371 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:29.746346 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f940b017-c539-41e8-be1a-c61f0f9ae25d-kube-rbac-proxy-cm podName:f940b017-c539-41e8-be1a-c61f0f9ae25d nodeName:}" failed. No retries permitted until 2026-09-16 13:31:31.746323903 +0000 UTC m=+52.790626853 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f940b017-c539-41e8-be1a-c61f0f9ae25d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hvzrx" (UID: "f940b017-c539-41e8-be1a-c61f0f9ae25d") : configmap "kube-rbac-proxy" not found Sep 16 13:31:29.746512 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:29.746372 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f940b017-c539-41e8-be1a-c61f0f9ae25d-insights-runtime-extractor-tls podName:f940b017-c539-41e8-be1a-c61f0f9ae25d nodeName:}" failed. No retries permitted until 2026-09-16 13:31:31.746362453 +0000 UTC m=+52.790665399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f940b017-c539-41e8-be1a-c61f0f9ae25d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvzrx" (UID: "f940b017-c539-41e8-be1a-c61f0f9ae25d") : secret "insights-runtime-extractor-tls" not found Sep 16 13:31:30.271744 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:30.271710 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t9qw9_24582309-5bb1-4d3d-8a18-a1d7803f287a/dns-node-resolver/0.log" Sep 16 13:31:30.870827 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:30.870802 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mcc87_e3cad145-80e4-4662-a1d8-e173c3cddf03/node-ca/0.log" Sep 16 13:31:31.638311 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:31.638277 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-6hxrk" event={"ID":"96770fb2-49ff-487d-831c-4579e2b91dc4","Type":"ContainerStarted","Data":"31b39d9c4433e969a112e1a52f0606bf18e1241f2d4068834518884b7e0a57c5"} Sep 16 13:31:31.656556 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:31.656514 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-6hxrk" podStartSLOduration=2.095849851 podStartE2EDuration="5.656499223s" totalImagesPullingTime="3.560649372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:31.656237864 +0000 UTC m=+52.700540830" watchObservedRunningTime="2026-09-16 13:31:31.656499223 +0000 UTC m=+52.700802188" Sep 16 13:31:31.672081 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:31.672058 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sfzx5_d5ab9c39-b9a9-42e1-8175-45c414ec2de9/serve-healthcheck-canary/0.log" Sep 16 13:31:31.764288 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:31.764261 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f940b017-c539-41e8-be1a-c61f0f9ae25d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvzrx\" (UID: \"f940b017-c539-41e8-be1a-c61f0f9ae25d\") " pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:31.764438 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:31.764372 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f940b017-c539-41e8-be1a-c61f0f9ae25d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvzrx\" (UID: \"f940b017-c539-41e8-be1a-c61f0f9ae25d\") " pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:31.764541 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:31.764521 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:31:31.764616 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:31.764604 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f940b017-c539-41e8-be1a-c61f0f9ae25d-insights-runtime-extractor-tls podName:f940b017-c539-41e8-be1a-c61f0f9ae25d nodeName:}" failed. No retries permitted until 2026-09-16 13:31:35.764582918 +0000 UTC m=+56.808885886 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f940b017-c539-41e8-be1a-c61f0f9ae25d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvzrx" (UID: "f940b017-c539-41e8-be1a-c61f0f9ae25d") : secret "insights-runtime-extractor-tls" not found Sep 16 13:31:31.764871 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:31.764854 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f940b017-c539-41e8-be1a-c61f0f9ae25d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvzrx\" (UID: \"f940b017-c539-41e8-be1a-c61f0f9ae25d\") " pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:31.816941 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:31.816910 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" Sep 16 13:31:31.817077 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:31.816983 2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" Sep 16 13:31:31.817314 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:31.817303 2562 scope.go:117] "RemoveContainer" containerID="fc4b744a9e8d661e945c7f5d1c13ed1437f2363d75c049ce19803fd501cd2381" Sep 16 13:31:31.817503 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:31.817482 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-4xmxv_openshift-console-operator(c7828efb-6be7-4f7b-986f-645a2370163a)\"" pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" podUID="c7828efb-6be7-4f7b-986f-645a2370163a" Sep 16 13:31:35.798556 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:35.798524 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f940b017-c539-41e8-be1a-c61f0f9ae25d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvzrx\" (UID: \"f940b017-c539-41e8-be1a-c61f0f9ae25d\") " pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:35.799039 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:35.798632 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 13:31:35.799039 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:35.798684 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f940b017-c539-41e8-be1a-c61f0f9ae25d-insights-runtime-extractor-tls podName:f940b017-c539-41e8-be1a-c61f0f9ae25d nodeName:}" failed. No retries permitted until 2026-09-16 13:31:43.798670628 +0000 UTC m=+64.842973572 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f940b017-c539-41e8-be1a-c61f0f9ae25d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvzrx" (UID: "f940b017-c539-41e8-be1a-c61f0f9ae25d") : secret "insights-runtime-extractor-tls" not found Sep 16 13:31:37.310502 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:37.310473 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d3da9f2-755a-47ea-b78c-2660c79e534c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-pw5cd\" (UID: \"0d3da9f2-755a-47ea-b78c-2660c79e534c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pw5cd" Sep 16 13:31:37.310502 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:37.310505 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-metrics-certs\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:37.310877 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:37.310532 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbc67ede-7ee2-4ac0-b256-952341815e01-service-ca-bundle\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:37.310877 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:37.310612 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 13:31:37.310877 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:37.310631 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbc67ede-7ee2-4ac0-b256-952341815e01-service-ca-bundle podName:fbc67ede-7ee2-4ac0-b256-952341815e01 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:53.310613661 +0000 UTC m=+74.354916606 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fbc67ede-7ee2-4ac0-b256-952341815e01-service-ca-bundle") pod "router-default-858dfcf476-rkhcs" (UID: "fbc67ede-7ee2-4ac0-b256-952341815e01") : configmap references non-existent config key: service-ca.crt Sep 16 13:31:37.310877 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:37.310669 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3da9f2-755a-47ea-b78c-2660c79e534c-cluster-monitoring-operator-tls podName:0d3da9f2-755a-47ea-b78c-2660c79e534c nodeName:}" failed. No retries permitted until 2026-09-16 13:31:53.310656727 +0000 UTC m=+74.354959703 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0d3da9f2-755a-47ea-b78c-2660c79e534c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-pw5cd" (UID: "0d3da9f2-755a-47ea-b78c-2660c79e534c") : secret "cluster-monitoring-operator-tls" not found Sep 16 13:31:37.310877 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:37.310612 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 13:31:37.310877 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:31:37.310752 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-metrics-certs podName:fbc67ede-7ee2-4ac0-b256-952341815e01 nodeName:}" failed. No retries permitted until 2026-09-16 13:31:53.310740621 +0000 UTC m=+74.355043565 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-metrics-certs") pod "router-default-858dfcf476-rkhcs" (UID: "fbc67ede-7ee2-4ac0-b256-952341815e01") : secret "router-metrics-certs-default" not found Sep 16 13:31:37.310877 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:37.310696 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c074fd6-74bd-4ef3-a90f-a620fe95962c-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-s2cmm\" (UID: \"6c074fd6-74bd-4ef3-a90f-a620fe95962c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s2cmm" Sep 16 13:31:37.312950 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:37.312935 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c074fd6-74bd-4ef3-a90f-a620fe95962c-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-s2cmm\" (UID: \"6c074fd6-74bd-4ef3-a90f-a620fe95962c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s2cmm" Sep 16 13:31:37.604013 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:37.603944 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s2cmm" Sep 16 13:31:37.872598 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:37.872575 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s2cmm"] Sep 16 13:31:38.544799 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:38.544773 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2k2tg" Sep 16 13:31:38.662191 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:38.662164 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s2cmm" event={"ID":"6c074fd6-74bd-4ef3-a90f-a620fe95962c","Type":"ContainerStarted","Data":"d64b3f5166300976c6fc0e0ffec7c857187b67e5f943c49c2f72b0c6a8e18ba4"} Sep 16 13:31:40.669870 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:40.669829 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s2cmm" event={"ID":"6c074fd6-74bd-4ef3-a90f-a620fe95962c","Type":"ContainerStarted","Data":"dbc6686372514d52c5b0d265bc582a6ff6bdef4e717a07e40f19d288f580b497"} Sep 16 13:31:40.670229 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:40.669878 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s2cmm" event={"ID":"6c074fd6-74bd-4ef3-a90f-a620fe95962c","Type":"ContainerStarted","Data":"c8c6198a978254d3c833347f4264ec6c890c70a81dde511ad88c4ad332608c45"} Sep 16 13:31:40.687549 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:40.687496 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s2cmm" podStartSLOduration=17.915180141 podStartE2EDuration="19.687479462s" totalImagesPullingTime="1.772299321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:40.687252449 +0000 UTC m=+61.731555415" watchObservedRunningTime="2026-09-16 13:31:40.687479462 +0000 UTC m=+61.731782446" Sep 16 13:31:43.865785 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:43.865752 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f940b017-c539-41e8-be1a-c61f0f9ae25d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvzrx\" (UID: \"f940b017-c539-41e8-be1a-c61f0f9ae25d\") " pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:43.868333 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:43.868308 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f940b017-c539-41e8-be1a-c61f0f9ae25d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvzrx\" (UID: \"f940b017-c539-41e8-be1a-c61f0f9ae25d\") " pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:43.944043 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:43.944021 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7rd5w\"" Sep 16 13:31:43.952291 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:43.952270 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hvzrx" Sep 16 13:31:44.219259 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:44.219231 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hvzrx"] Sep 16 13:31:44.222049 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:31:44.222010 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf940b017_c539_41e8_be1a_c61f0f9ae25d.slice/crio-d11b61922d2e0752b36b71949646218037ce3219475eca7ed578e92f4216658b WatchSource:0}: Error finding container d11b61922d2e0752b36b71949646218037ce3219475eca7ed578e92f4216658b: Status 404 returned error can't find the container with id d11b61922d2e0752b36b71949646218037ce3219475eca7ed578e92f4216658b Sep 16 13:31:44.683723 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:44.683688 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hvzrx" event={"ID":"f940b017-c539-41e8-be1a-c61f0f9ae25d","Type":"ContainerStarted","Data":"2d2eaea6371254746b639e50c3f2908917617d3b29400377827da5e350d01db2"} Sep 16 13:31:44.683864 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:44.683747 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hvzrx" event={"ID":"f940b017-c539-41e8-be1a-c61f0f9ae25d","Type":"ContainerStarted","Data":"d11b61922d2e0752b36b71949646218037ce3219475eca7ed578e92f4216658b"} Sep 16 13:31:45.279096 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.279069 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-original-pull-secret\") pod \"global-pull-secret-syncer-5s6pv\" (UID: \"2a102e1b-9b9b-45f3-9e7e-33794568bdb9\") " pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:31:45.279450 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.279163 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/390ee01e-b850-45d2-b21d-c6b002a1c0ff-metrics-tls\") pod \"dns-default-mxnjp\" (UID: \"390ee01e-b850-45d2-b21d-c6b002a1c0ff\") " pod="openshift-dns/dns-default-mxnjp" Sep 16 13:31:45.279450 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.279199 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/816a366f-b67b-4679-a323-ab36289dc4b6-metrics-certs\") pod \"network-metrics-daemon-hm6lg\" (UID: \"816a366f-b67b-4679-a323-ab36289dc4b6\") " pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:31:45.279450 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.279220 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-tls\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:45.281592 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.281564 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/390ee01e-b850-45d2-b21d-c6b002a1c0ff-metrics-tls\") pod \"dns-default-mxnjp\" (UID: \"390ee01e-b850-45d2-b21d-c6b002a1c0ff\") " pod="openshift-dns/dns-default-mxnjp" Sep 16 13:31:45.281730 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.281710 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-tls\") pod \"image-registry-6449dbdc57-jc64r\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:45.281914 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.281898 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 13:31:45.281998 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.281981 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 13:31:45.292212 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.292189 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a102e1b-9b9b-45f3-9e7e-33794568bdb9-original-pull-secret\") pod \"global-pull-secret-syncer-5s6pv\" (UID: \"2a102e1b-9b9b-45f3-9e7e-33794568bdb9\") " pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:31:45.292291 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.292190 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/816a366f-b67b-4679-a323-ab36289dc4b6-metrics-certs\") pod \"network-metrics-daemon-hm6lg\" (UID: \"816a366f-b67b-4679-a323-ab36289dc4b6\") " pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:31:45.379770 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.379734 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzvrt\" (UniqueName: \"kubernetes.io/projected/205445dc-c385-470f-bc60-dfa12d107297-kube-api-access-nzvrt\") pod \"network-check-target-vg66w\" (UID: \"205445dc-c385-470f-bc60-dfa12d107297\") " pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:31:45.381981 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.381965 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzvrt\" (UniqueName: \"kubernetes.io/projected/205445dc-c385-470f-bc60-dfa12d107297-kube-api-access-nzvrt\") pod \"network-check-target-vg66w\" (UID: \"205445dc-c385-470f-bc60-dfa12d107297\") " pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:31:45.407289 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.407264 2562 scope.go:117] "RemoveContainer" containerID="fc4b744a9e8d661e945c7f5d1c13ed1437f2363d75c049ce19803fd501cd2381" Sep 16 13:31:45.448161 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.448144 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s4fpm\"" Sep 16 13:31:45.455523 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.455502 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:45.459880 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.459864 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t4m6m\"" Sep 16 13:31:45.468104 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.468082 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mxnjp" Sep 16 13:31:45.522391 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.522357 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s6pv" Sep 16 13:31:45.524441 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.523913 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4nxp8\"" Sep 16 13:31:45.528613 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.527295 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:31:45.530889 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.530870 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4n8lp\"" Sep 16 13:31:45.538467 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.538445 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hm6lg" Sep 16 13:31:45.689420 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.689339 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4xmxv_c7828efb-6be7-4f7b-986f-645a2370163a/console-operator/1.log" Sep 16 13:31:45.689538 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.689418 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" event={"ID":"c7828efb-6be7-4f7b-986f-645a2370163a","Type":"ContainerStarted","Data":"4e35d783a3aac39b861d190b92906bf3566781c67459a838775ebcb18e537526"} Sep 16 13:31:45.689856 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.689840 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" Sep 16 13:31:45.691487 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.691466 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hvzrx" event={"ID":"f940b017-c539-41e8-be1a-c61f0f9ae25d","Type":"ContainerStarted","Data":"fede33c1a61e73fda17cd5b7feab234ecfc9199ca3649ea47eca5c9936d29d0c"} Sep 16 13:31:45.709946 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.709317 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" podStartSLOduration=20.00124118 podStartE2EDuration="24.709301893s" totalImagesPullingTime="4.708060713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:45.708879475 +0000 UTC m=+66.753182458" watchObservedRunningTime="2026-09-16 13:31:45.709301893 +0000 UTC m=+66.753604860" Sep 16 13:31:45.729475 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.729457 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6449dbdc57-jc64r"] Sep 16 13:31:45.732297 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:31:45.732274 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c0f9ae_bb5d_4621_b804_29b5a3a88f6c.slice/crio-e95d5e468f7201f5941c578b017ba4bcf9ebdb6e985988b65a395dbadbfe80b8 WatchSource:0}: Error finding container e95d5e468f7201f5941c578b017ba4bcf9ebdb6e985988b65a395dbadbfe80b8: Status 404 returned error can't find the container with id e95d5e468f7201f5941c578b017ba4bcf9ebdb6e985988b65a395dbadbfe80b8 Sep 16 13:31:45.754973 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.754956 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mxnjp"] Sep 16 13:31:45.756760 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:31:45.756742 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390ee01e_b850_45d2_b21d_c6b002a1c0ff.slice/crio-8beca09c28cf0295b46b8a0e931b32be21dba8ee615b7008352a2c82af087af7 WatchSource:0}: Error finding container 8beca09c28cf0295b46b8a0e931b32be21dba8ee615b7008352a2c82af087af7: Status 404 returned error can't find the container with id 8beca09c28cf0295b46b8a0e931b32be21dba8ee615b7008352a2c82af087af7 Sep 16 13:31:45.815729 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.815605 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vg66w"] Sep 16 13:31:45.816952 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.816932 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5s6pv"] Sep 16 13:31:45.818216 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:31:45.818191 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a102e1b_9b9b_45f3_9e7e_33794568bdb9.slice/crio-b9239de814b96cb5761e8a84049eeb850a8a17efd965959affb579d17f95bd7c WatchSource:0}: Error finding container b9239de814b96cb5761e8a84049eeb850a8a17efd965959affb579d17f95bd7c: Status 404 returned error can't find the container with id b9239de814b96cb5761e8a84049eeb850a8a17efd965959affb579d17f95bd7c Sep 16 13:31:45.819216 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:31:45.819194 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod205445dc_c385_470f_bc60_dfa12d107297.slice/crio-02e8f1c0aa08793caec27d181657653e665136f92fff50174e25d2e750583bb6 WatchSource:0}: Error finding container 02e8f1c0aa08793caec27d181657653e665136f92fff50174e25d2e750583bb6: Status 404 returned error can't find the container with id 02e8f1c0aa08793caec27d181657653e665136f92fff50174e25d2e750583bb6 Sep 16 13:31:45.825670 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:45.825642 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hm6lg"] Sep 16 13:31:45.829502 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:31:45.829482 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816a366f_b67b_4679_a323_ab36289dc4b6.slice/crio-b8994f9996e3bf007e746ef51a81fa7fe015378bf2b927852e10066558400886 WatchSource:0}: Error finding container b8994f9996e3bf007e746ef51a81fa7fe015378bf2b927852e10066558400886: Status 404 returned error can't find the container with id b8994f9996e3bf007e746ef51a81fa7fe015378bf2b927852e10066558400886 Sep 16 13:31:46.316669 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:46.316638 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-4xmxv" Sep 16 13:31:46.699489 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:46.699454 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mxnjp" event={"ID":"390ee01e-b850-45d2-b21d-c6b002a1c0ff","Type":"ContainerStarted","Data":"8beca09c28cf0295b46b8a0e931b32be21dba8ee615b7008352a2c82af087af7"} Sep 16 13:31:46.702213 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:46.701913 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" event={"ID":"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c","Type":"ContainerStarted","Data":"d6cbc34decf12e65eaaff6b44bc1eb675f628b30e6127b45cccbeddce2dfdb85"} Sep 16 13:31:46.702213 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:46.701956 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" event={"ID":"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c","Type":"ContainerStarted","Data":"e95d5e468f7201f5941c578b017ba4bcf9ebdb6e985988b65a395dbadbfe80b8"} Sep 16 13:31:46.702213 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:46.702170 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:31:46.704145 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:46.704125 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vg66w" event={"ID":"205445dc-c385-470f-bc60-dfa12d107297","Type":"ContainerStarted","Data":"14b62f72b954eb90791a15a590f414e1afa6d34018c64fb8f226c9ff0ed7b1fa"} Sep 16 13:31:46.704245 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:46.704163 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vg66w" event={"ID":"205445dc-c385-470f-bc60-dfa12d107297","Type":"ContainerStarted","Data":"02e8f1c0aa08793caec27d181657653e665136f92fff50174e25d2e750583bb6"} Sep 16 13:31:46.704245 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:46.704236 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:31:46.706230 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:46.706202 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5s6pv" event={"ID":"2a102e1b-9b9b-45f3-9e7e-33794568bdb9","Type":"ContainerStarted","Data":"b9239de814b96cb5761e8a84049eeb850a8a17efd965959affb579d17f95bd7c"} Sep 16 13:31:46.707826 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:46.707805 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hm6lg" event={"ID":"816a366f-b67b-4679-a323-ab36289dc4b6","Type":"ContainerStarted","Data":"b8994f9996e3bf007e746ef51a81fa7fe015378bf2b927852e10066558400886"} Sep 16 13:31:46.723398 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:46.723329 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" podStartSLOduration=67.723312293 podStartE2EDuration="1m7.723312293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:30:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:46.721992513 +0000 UTC m=+67.766295501" watchObservedRunningTime="2026-09-16 13:31:46.723312293 +0000 UTC m=+67.767615261" Sep 16 13:31:46.738493 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:46.737768 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vg66w" podStartSLOduration=67.737750363 podStartE2EDuration="1m7.737750363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:30:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:46.736645283 +0000 UTC m=+67.780948251" watchObservedRunningTime="2026-09-16 13:31:46.737750363 +0000 UTC m=+67.782053332" Sep 16 13:31:48.717878 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:48.717836 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hvzrx" event={"ID":"f940b017-c539-41e8-be1a-c61f0f9ae25d","Type":"ContainerStarted","Data":"59cd15f6e58b72699a76256932d9afe3cb9315b7d8da5fbd6a3d79fb3743c139"} Sep 16 13:31:48.720181 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:48.720141 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hm6lg" event={"ID":"816a366f-b67b-4679-a323-ab36289dc4b6","Type":"ContainerStarted","Data":"79c6d6a24fc3853091c8dd8dde22094696521bcb536f6db5304d861653493951"} Sep 16 13:31:48.720321 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:48.720200 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hm6lg" event={"ID":"816a366f-b67b-4679-a323-ab36289dc4b6","Type":"ContainerStarted","Data":"3debf12495011f2263904b3cd42a382211e9f62cfd5ac27860196ac02fcfe055"} Sep 16 13:31:48.722240 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:48.722211 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mxnjp" event={"ID":"390ee01e-b850-45d2-b21d-c6b002a1c0ff","Type":"ContainerStarted","Data":"68079164af895ed740ffba2e0eca1f2da49165b7fa5daffdcba5977414400d18"} Sep 16 13:31:48.722342 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:48.722262 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mxnjp" event={"ID":"390ee01e-b850-45d2-b21d-c6b002a1c0ff","Type":"ContainerStarted","Data":"a39e9adbfed9f3468a5451f173ce479aa5d84dbf315d69ff6fbd8095db3a487c"} Sep 16 13:31:48.722342 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:48.722306 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mxnjp" Sep 16 13:31:48.739428 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:48.739332 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hvzrx" podStartSLOduration=17.989055161 podStartE2EDuration="21.739317338s" totalImagesPullingTime="3.750262177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:31:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:48.738569054 +0000 UTC m=+69.782872046" watchObservedRunningTime="2026-09-16 13:31:48.739317338 +0000 UTC m=+69.783620304" Sep 16 13:31:48.758004 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:48.757963 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mxnjp" podStartSLOduration=33.443373095 podStartE2EDuration="35.757949933s" totalImagesPullingTime="2.314576838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:31:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:48.757438233 +0000 UTC m=+69.801741223" watchObservedRunningTime="2026-09-16 13:31:48.757949933 +0000 UTC m=+69.802252893" Sep 16 13:31:48.775022 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:48.774977 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hm6lg" podStartSLOduration=67.533601804 podStartE2EDuration="1m9.774962354s" totalImagesPullingTime="2.24136055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:48.774038444 +0000 UTC m=+69.818341432" watchObservedRunningTime="2026-09-16 13:31:48.774962354 +0000 UTC m=+69.819265319" Sep 16 13:31:50.732311 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:50.732274 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5s6pv" event={"ID":"2a102e1b-9b9b-45f3-9e7e-33794568bdb9","Type":"ContainerStarted","Data":"1ac30b96199b0e462c0fcc18ad521c1e5c9bd4d7f6a0700255aa9e1ddb77813d"} Sep 16 13:31:50.750444 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:50.750348 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5s6pv" podStartSLOduration=67.068556637 podStartE2EDuration="1m11.750333522s" totalImagesPullingTime="4.681776885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:50.749509247 +0000 UTC m=+71.793812232" watchObservedRunningTime="2026-09-16 13:31:50.750333522 +0000 UTC m=+71.794636485" Sep 16 13:31:53.349202 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:53.349165 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d3da9f2-755a-47ea-b78c-2660c79e534c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-pw5cd\" (UID: \"0d3da9f2-755a-47ea-b78c-2660c79e534c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pw5cd" Sep 16 13:31:53.349202 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:53.349205 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-metrics-certs\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:53.349797 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:53.349236 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbc67ede-7ee2-4ac0-b256-952341815e01-service-ca-bundle\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:53.349898 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:53.349880 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbc67ede-7ee2-4ac0-b256-952341815e01-service-ca-bundle\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:53.351699 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:53.351673 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d3da9f2-755a-47ea-b78c-2660c79e534c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-pw5cd\" (UID: \"0d3da9f2-755a-47ea-b78c-2660c79e534c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pw5cd" Sep 16 13:31:53.351861 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:53.351840 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc67ede-7ee2-4ac0-b256-952341815e01-metrics-certs\") pod \"router-default-858dfcf476-rkhcs\" (UID: \"fbc67ede-7ee2-4ac0-b256-952341815e01\") " pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:53.493309 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:53.493272 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tg274\"" Sep 16 13:31:53.500692 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:53.500667 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pw5cd" Sep 16 13:31:53.549544 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:53.549120 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6lms4\"" Sep 16 13:31:53.556986 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:53.556964 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:53.769280 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:53.769258 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-pw5cd"] Sep 16 13:31:53.771453 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:31:53.771429 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3da9f2_755a_47ea_b78c_2660c79e534c.slice/crio-acaf4c0f85958d2003f2e123ba94f35fe826876c56c86aa84bc62e46a6f832b7 WatchSource:0}: Error finding container acaf4c0f85958d2003f2e123ba94f35fe826876c56c86aa84bc62e46a6f832b7: Status 404 returned error can't find the container with id acaf4c0f85958d2003f2e123ba94f35fe826876c56c86aa84bc62e46a6f832b7 Sep 16 13:31:53.832263 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:53.832208 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-858dfcf476-rkhcs"] Sep 16 13:31:53.834045 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:31:53.834025 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc67ede_7ee2_4ac0_b256_952341815e01.slice/crio-98365e46051d5e6573a97c425ff2aef621a556a0b4f801d9c6856522ffd54e27 WatchSource:0}: Error finding container 98365e46051d5e6573a97c425ff2aef621a556a0b4f801d9c6856522ffd54e27: Status 404 returned error can't find the container with id 98365e46051d5e6573a97c425ff2aef621a556a0b4f801d9c6856522ffd54e27 Sep 16 13:31:54.291552 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.291520 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-6pfrf"] Sep 16 13:31:54.296666 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.296642 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-ww7v7"] Sep 16 13:31:54.296808 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.296760 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6pfrf" Sep 16 13:31:54.298630 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.298612 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-ww7v7" Sep 16 13:31:54.307067 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.307048 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 16 13:31:54.307625 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.307612 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 16 13:31:54.307902 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.307889 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-f5m9c\"" Sep 16 13:31:54.308937 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.308920 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-6hr5f\"" Sep 16 13:31:54.313081 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.313065 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 16 13:31:54.313295 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.313284 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 16 13:31:54.322042 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.322023 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-ww7v7"] Sep 16 13:31:54.323011 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.322994 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-6pfrf"] Sep 16 13:31:54.458507 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.458477 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/186f4dda-b214-4495-b832-c9866c7dabb8-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-6pfrf\" (UID: \"186f4dda-b214-4495-b832-c9866c7dabb8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6pfrf" Sep 16 13:31:54.458858 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.458519 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tw8v\" (UniqueName: \"kubernetes.io/projected/e8e0f81e-1e35-49fd-8725-65e1aecac15d-kube-api-access-2tw8v\") pod \"downloads-76f99584f4-ww7v7\" (UID: \"e8e0f81e-1e35-49fd-8725-65e1aecac15d\") " pod="openshift-console/downloads-76f99584f4-ww7v7" Sep 16 13:31:54.458858 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.458545 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/186f4dda-b214-4495-b832-c9866c7dabb8-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-6pfrf\" (UID: \"186f4dda-b214-4495-b832-c9866c7dabb8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6pfrf" Sep 16 13:31:54.558998 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.558928 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tw8v\" (UniqueName: \"kubernetes.io/projected/e8e0f81e-1e35-49fd-8725-65e1aecac15d-kube-api-access-2tw8v\") pod \"downloads-76f99584f4-ww7v7\" (UID: \"e8e0f81e-1e35-49fd-8725-65e1aecac15d\") " pod="openshift-console/downloads-76f99584f4-ww7v7" Sep 16 13:31:54.558998 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.558981 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/186f4dda-b214-4495-b832-c9866c7dabb8-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-6pfrf\" (UID: \"186f4dda-b214-4495-b832-c9866c7dabb8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6pfrf" Sep 16 13:31:54.559188 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.559042 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/186f4dda-b214-4495-b832-c9866c7dabb8-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-6pfrf\" (UID: \"186f4dda-b214-4495-b832-c9866c7dabb8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6pfrf" Sep 16 13:31:54.559772 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.559743 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/186f4dda-b214-4495-b832-c9866c7dabb8-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-6pfrf\" (UID: \"186f4dda-b214-4495-b832-c9866c7dabb8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6pfrf" Sep 16 13:31:54.561572 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.561552 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/186f4dda-b214-4495-b832-c9866c7dabb8-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-6pfrf\" (UID: \"186f4dda-b214-4495-b832-c9866c7dabb8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6pfrf" Sep 16 13:31:54.578630 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.578609 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tw8v\" (UniqueName: \"kubernetes.io/projected/e8e0f81e-1e35-49fd-8725-65e1aecac15d-kube-api-access-2tw8v\") pod \"downloads-76f99584f4-ww7v7\" (UID: \"e8e0f81e-1e35-49fd-8725-65e1aecac15d\") " pod="openshift-console/downloads-76f99584f4-ww7v7" Sep 16 13:31:54.606478 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.606458 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6pfrf" Sep 16 13:31:54.612355 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.612332 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-ww7v7" Sep 16 13:31:54.750587 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.750546 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pw5cd" event={"ID":"0d3da9f2-755a-47ea-b78c-2660c79e534c","Type":"ContainerStarted","Data":"acaf4c0f85958d2003f2e123ba94f35fe826876c56c86aa84bc62e46a6f832b7"} Sep 16 13:31:54.753036 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.752563 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-858dfcf476-rkhcs" event={"ID":"fbc67ede-7ee2-4ac0-b256-952341815e01","Type":"ContainerStarted","Data":"06f36a157a9a97449612b291720a3741f2f07349d10da109de6f015625efe6e6"} Sep 16 13:31:54.753036 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.752618 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-858dfcf476-rkhcs" event={"ID":"fbc67ede-7ee2-4ac0-b256-952341815e01","Type":"ContainerStarted","Data":"98365e46051d5e6573a97c425ff2aef621a556a0b4f801d9c6856522ffd54e27"} Sep 16 13:31:54.777844 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.777800 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-858dfcf476-rkhcs" podStartSLOduration=33.777759497 podStartE2EDuration="33.777759497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:54.776877492 +0000 UTC m=+75.821180489" watchObservedRunningTime="2026-09-16 13:31:54.777759497 +0000 UTC m=+75.822062463" Sep 16 13:31:54.889921 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.889809 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-6pfrf"] Sep 16 13:31:54.892442 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:31:54.892411 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186f4dda_b214_4495_b832_c9866c7dabb8.slice/crio-b4ddf7e82d49f32294148bb2578e35397cd488979c78382d647d29f3479bf30b WatchSource:0}: Error finding container b4ddf7e82d49f32294148bb2578e35397cd488979c78382d647d29f3479bf30b: Status 404 returned error can't find the container with id b4ddf7e82d49f32294148bb2578e35397cd488979c78382d647d29f3479bf30b Sep 16 13:31:54.908230 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:54.908204 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-ww7v7"] Sep 16 13:31:54.911067 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:31:54.911040 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e0f81e_1e35_49fd_8725_65e1aecac15d.slice/crio-c3029838172ab2b993e4649aab71bb0da8eec497d143bf46b4c97f0c706d2fae WatchSource:0}: Error finding container c3029838172ab2b993e4649aab71bb0da8eec497d143bf46b4c97f0c706d2fae: Status 404 returned error can't find the container with id c3029838172ab2b993e4649aab71bb0da8eec497d143bf46b4c97f0c706d2fae Sep 16 13:31:55.558080 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:55.558037 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:55.560924 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:55.560903 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:55.758173 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:55.758134 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-ww7v7" event={"ID":"e8e0f81e-1e35-49fd-8725-65e1aecac15d","Type":"ContainerStarted","Data":"c3029838172ab2b993e4649aab71bb0da8eec497d143bf46b4c97f0c706d2fae"} Sep 16 13:31:55.759859 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:55.759831 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pw5cd" event={"ID":"0d3da9f2-755a-47ea-b78c-2660c79e534c","Type":"ContainerStarted","Data":"db9baaebe27bbc14ed2e317357445e776a99502ac35c7ea6b6e3a54c1fabfd8d"} Sep 16 13:31:55.761115 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:55.761089 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6pfrf" event={"ID":"186f4dda-b214-4495-b832-c9866c7dabb8","Type":"ContainerStarted","Data":"b4ddf7e82d49f32294148bb2578e35397cd488979c78382d647d29f3479bf30b"} Sep 16 13:31:55.761260 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:55.761242 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:55.763147 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:55.763129 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-858dfcf476-rkhcs" Sep 16 13:31:55.781332 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:55.781285 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-pw5cd" podStartSLOduration=32.895509627 podStartE2EDuration="34.78127322s" totalImagesPullingTime="1.885763593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:55.780164193 +0000 UTC m=+76.824467189" watchObservedRunningTime="2026-09-16 13:31:55.78127322 +0000 UTC m=+76.825576186" Sep 16 13:31:58.729484 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:58.729447 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mxnjp" Sep 16 13:31:58.778211 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:58.778168 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6pfrf" event={"ID":"186f4dda-b214-4495-b832-c9866c7dabb8","Type":"ContainerStarted","Data":"8af3b12b700b7d52b9fca09f6d725a5cc697ddeee2ab9aede1384f23b38d5e65"} Sep 16 13:31:58.796352 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:31:58.796211 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6pfrf" podStartSLOduration=1.589928684 podStartE2EDuration="4.796196717s" totalImagesPullingTime="3.206268033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:31:58.794857602 +0000 UTC m=+79.839160588" watchObservedRunningTime="2026-09-16 13:31:58.796196717 +0000 UTC m=+79.840499682" Sep 16 13:32:03.412971 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:03.412606 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-gtb5v"] Sep 16 13:32:03.415733 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:03.415706 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-gtb5v" Sep 16 13:32:03.418534 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:03.418303 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-f6pmt\"" Sep 16 13:32:03.419605 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:03.419588 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 16 13:32:03.419813 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:03.419787 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 16 13:32:03.419883 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:03.419834 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 16 13:32:03.422229 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:03.422204 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-gtb5v"] Sep 16 13:32:03.533414 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:03.533369 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lhkk\" (UniqueName: \"kubernetes.io/projected/cabf34ef-ee8c-4052-95e9-8f4fe6d550e9-kube-api-access-6lhkk\") pod \"prometheus-operator-68bcd7f987-gtb5v\" (UID: \"cabf34ef-ee8c-4052-95e9-8f4fe6d550e9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gtb5v" Sep 16 13:32:03.533605 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:03.533510 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cabf34ef-ee8c-4052-95e9-8f4fe6d550e9-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-gtb5v\" (UID: \"cabf34ef-ee8c-4052-95e9-8f4fe6d550e9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gtb5v" Sep 16 13:32:03.533605 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:03.533546 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cabf34ef-ee8c-4052-95e9-8f4fe6d550e9-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-gtb5v\" (UID: \"cabf34ef-ee8c-4052-95e9-8f4fe6d550e9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gtb5v" Sep 16 13:32:03.533605 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:03.533585 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cabf34ef-ee8c-4052-95e9-8f4fe6d550e9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-gtb5v\" (UID: \"cabf34ef-ee8c-4052-95e9-8f4fe6d550e9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gtb5v" Sep 16 13:32:03.634604 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:03.634573 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lhkk\" (UniqueName: \"kubernetes.io/projected/cabf34ef-ee8c-4052-95e9-8f4fe6d550e9-kube-api-access-6lhkk\") pod \"prometheus-operator-68bcd7f987-gtb5v\" (UID: \"cabf34ef-ee8c-4052-95e9-8f4fe6d550e9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gtb5v" Sep 16 13:32:03.634828 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:03.634635 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cabf34ef-ee8c-4052-95e9-8f4fe6d550e9-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-gtb5v\" (UID: \"cabf34ef-ee8c-4052-95e9-8f4fe6d550e9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gtb5v" Sep 16 13:32:03.634828 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:03.634663 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cabf34ef-ee8c-4052-95e9-8f4fe6d550e9-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-gtb5v\" (UID: \"cabf34ef-ee8c-4052-95e9-8f4fe6d550e9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gtb5v" Sep 16 13:32:03.634828 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:03.634694 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cabf34ef-ee8c-4052-95e9-8f4fe6d550e9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-gtb5v\" (UID: \"cabf34ef-ee8c-4052-95e9-8f4fe6d550e9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gtb5v" Sep 16 13:32:03.635249 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:03.635229 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cabf34ef-ee8c-4052-95e9-8f4fe6d550e9-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-gtb5v\" (UID: \"cabf34ef-ee8c-4052-95e9-8f4fe6d550e9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gtb5v" Sep 16 13:32:03.637099 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:03.637075 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cabf34ef-ee8c-4052-95e9-8f4fe6d550e9-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-gtb5v\" (UID: \"cabf34ef-ee8c-4052-95e9-8f4fe6d550e9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gtb5v" Sep 16 13:32:03.637234 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:03.637215 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cabf34ef-ee8c-4052-95e9-8f4fe6d550e9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-gtb5v\" (UID: \"cabf34ef-ee8c-4052-95e9-8f4fe6d550e9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gtb5v" Sep 16 13:32:03.643129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:03.643111 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lhkk\" (UniqueName: \"kubernetes.io/projected/cabf34ef-ee8c-4052-95e9-8f4fe6d550e9-kube-api-access-6lhkk\") pod \"prometheus-operator-68bcd7f987-gtb5v\" (UID: \"cabf34ef-ee8c-4052-95e9-8f4fe6d550e9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gtb5v" Sep 16 13:32:03.727070 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:03.727022 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-gtb5v" Sep 16 13:32:04.004781 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:04.004755 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-gtb5v"] Sep 16 13:32:04.006800 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:32:04.006769 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabf34ef_ee8c_4052_95e9_8f4fe6d550e9.slice/crio-fb723b457b49f6e14a62da41a9f8e3db8acf6330a06045d7c996cc74cda435e5 WatchSource:0}: Error finding container fb723b457b49f6e14a62da41a9f8e3db8acf6330a06045d7c996cc74cda435e5: Status 404 returned error can't find the container with id fb723b457b49f6e14a62da41a9f8e3db8acf6330a06045d7c996cc74cda435e5 Sep 16 13:32:04.806149 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:04.806115 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-gtb5v" event={"ID":"cabf34ef-ee8c-4052-95e9-8f4fe6d550e9","Type":"ContainerStarted","Data":"fb723b457b49f6e14a62da41a9f8e3db8acf6330a06045d7c996cc74cda435e5"} Sep 16 13:32:07.717224 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:07.717192 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:32:12.841837 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:12.841790 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-gtb5v" event={"ID":"cabf34ef-ee8c-4052-95e9-8f4fe6d550e9","Type":"ContainerStarted","Data":"27af5fbef0d6226326cdb128896408b8838d784f6fa5c1cf03200a50cdefac08"} Sep 16 13:32:12.842293 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:12.841862 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-gtb5v" event={"ID":"cabf34ef-ee8c-4052-95e9-8f4fe6d550e9","Type":"ContainerStarted","Data":"7a3d83eeea0cf5248dc06fb28f5c3ce494a9f8c88d7b7d536dc8c1d4823bfea5"} Sep 16 13:32:12.843711 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:12.843686 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-ww7v7" event={"ID":"e8e0f81e-1e35-49fd-8725-65e1aecac15d","Type":"ContainerStarted","Data":"ff978746af5e6f47eff2ce542e9f2e028a5c44eb7068cca184163982d7bcc3da"} Sep 16 13:32:12.843938 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:12.843912 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-ww7v7" Sep 16 13:32:12.845460 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:12.845435 2562 patch_prober.go:28] interesting pod/downloads-76f99584f4-ww7v7 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.20:8080/\": dial tcp 10.132.0.20:8080: connect: connection refused" start-of-body= Sep 16 13:32:12.845554 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:12.845490 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-76f99584f4-ww7v7" podUID="e8e0f81e-1e35-49fd-8725-65e1aecac15d" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.20:8080/\": dial tcp 10.132.0.20:8080: connect: connection refused" Sep 16 13:32:12.864062 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:12.864024 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-gtb5v" podStartSLOduration=1.542435958 podStartE2EDuration="9.864010981s" totalImagesPullingTime="8.321575023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:32:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:32:12.863252296 +0000 UTC m=+93.907555285" watchObservedRunningTime="2026-09-16 13:32:12.864010981 +0000 UTC m=+93.908313947" Sep 16 13:32:13.864079 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:13.864046 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-ww7v7" Sep 16 13:32:13.896902 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:13.896842 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-ww7v7" podStartSLOduration=2.452266925 podStartE2EDuration="19.896821122s" totalImagesPullingTime="17.444554197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:32:12.883059053 +0000 UTC m=+93.927362032" watchObservedRunningTime="2026-09-16 13:32:13.896821122 +0000 UTC m=+94.941124089" Sep 16 13:32:14.858659 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:14.858626 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jttjb"] Sep 16 13:32:14.901169 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:14.901140 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:14.904079 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:14.904008 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 16 13:32:14.904211 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:14.904187 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-txljv\"" Sep 16 13:32:14.904264 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:14.904215 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 16 13:32:14.904741 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:14.904715 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 16 13:32:14.930624 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:14.930403 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd7rd\" (UniqueName: \"kubernetes.io/projected/624e58c2-824b-4b27-8f19-56f047656011-kube-api-access-pd7rd\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:14.930624 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:14.930465 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/624e58c2-824b-4b27-8f19-56f047656011-node-exporter-tls\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:14.930624 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:14.930514 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/624e58c2-824b-4b27-8f19-56f047656011-sys\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:14.930624 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:14.930539 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/624e58c2-824b-4b27-8f19-56f047656011-root\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:14.930624 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:14.930567 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/624e58c2-824b-4b27-8f19-56f047656011-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:14.930624 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:14.930591 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/624e58c2-824b-4b27-8f19-56f047656011-node-exporter-accelerators-collector-config\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:14.930995 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:14.930969 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/624e58c2-824b-4b27-8f19-56f047656011-metrics-client-ca\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:14.931095 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:14.931031 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/624e58c2-824b-4b27-8f19-56f047656011-node-exporter-textfile\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:14.931095 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:14.931059 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/624e58c2-824b-4b27-8f19-56f047656011-node-exporter-wtmp\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:15.031791 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:15.031756 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/624e58c2-824b-4b27-8f19-56f047656011-node-exporter-textfile\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:15.031791 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:15.031800 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/624e58c2-824b-4b27-8f19-56f047656011-node-exporter-wtmp\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:15.032021 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:15.031935 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd7rd\" (UniqueName: \"kubernetes.io/projected/624e58c2-824b-4b27-8f19-56f047656011-kube-api-access-pd7rd\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:15.032021 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:15.031954 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/624e58c2-824b-4b27-8f19-56f047656011-node-exporter-wtmp\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:15.032021 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:15.031979 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/624e58c2-824b-4b27-8f19-56f047656011-node-exporter-tls\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:15.032181 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:15.032040 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/624e58c2-824b-4b27-8f19-56f047656011-sys\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:15.032181 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:15.032067 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/624e58c2-824b-4b27-8f19-56f047656011-root\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:15.032181 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:15.032098 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/624e58c2-824b-4b27-8f19-56f047656011-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:15.032181 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:15.032124 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/624e58c2-824b-4b27-8f19-56f047656011-node-exporter-accelerators-collector-config\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:15.032181 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:15.032116 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/624e58c2-824b-4b27-8f19-56f047656011-node-exporter-textfile\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:15.032181 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:15.032148 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/624e58c2-824b-4b27-8f19-56f047656011-metrics-client-ca\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:15.032484 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:15.032185 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/624e58c2-824b-4b27-8f19-56f047656011-root\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:15.032484 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:32:15.032261 2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 16 13:32:15.032484 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:32:15.032314 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/624e58c2-824b-4b27-8f19-56f047656011-node-exporter-tls podName:624e58c2-824b-4b27-8f19-56f047656011 nodeName:}" failed. No retries permitted until 2026-09-16 13:32:15.532295821 +0000 UTC m=+96.576598778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/624e58c2-824b-4b27-8f19-56f047656011-node-exporter-tls") pod "node-exporter-jttjb" (UID: "624e58c2-824b-4b27-8f19-56f047656011") : secret "node-exporter-tls" not found Sep 16 13:32:15.032633 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:15.032555 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/624e58c2-824b-4b27-8f19-56f047656011-sys\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:15.032698 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:15.032678 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/624e58c2-824b-4b27-8f19-56f047656011-metrics-client-ca\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:15.032977 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:15.032957 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/624e58c2-824b-4b27-8f19-56f047656011-node-exporter-accelerators-collector-config\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:15.034890 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:15.034868 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/624e58c2-824b-4b27-8f19-56f047656011-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:15.050999 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:15.050977 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd7rd\" (UniqueName: \"kubernetes.io/projected/624e58c2-824b-4b27-8f19-56f047656011-kube-api-access-pd7rd\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:15.536781 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:15.536741 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/624e58c2-824b-4b27-8f19-56f047656011-node-exporter-tls\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:15.539552 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:15.539521 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/624e58c2-824b-4b27-8f19-56f047656011-node-exporter-tls\") pod \"node-exporter-jttjb\" (UID: \"624e58c2-824b-4b27-8f19-56f047656011\") " pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:15.812400 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:15.812298 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jttjb" Sep 16 13:32:15.849301 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:32:15.849270 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod624e58c2_824b_4b27_8f19_56f047656011.slice/crio-641d1a67b98aec47402fe83c8f43dfcf4903a2eb300bb07d187dec8d2bcb3a25 WatchSource:0}: Error finding container 641d1a67b98aec47402fe83c8f43dfcf4903a2eb300bb07d187dec8d2bcb3a25: Status 404 returned error can't find the container with id 641d1a67b98aec47402fe83c8f43dfcf4903a2eb300bb07d187dec8d2bcb3a25 Sep 16 13:32:15.858676 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:15.858643 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jttjb" event={"ID":"624e58c2-824b-4b27-8f19-56f047656011","Type":"ContainerStarted","Data":"641d1a67b98aec47402fe83c8f43dfcf4903a2eb300bb07d187dec8d2bcb3a25"} Sep 16 13:32:16.344356 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.344320 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6449dbdc57-jc64r"] Sep 16 13:32:16.765547 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.765466 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7d5668779d-z7sst"] Sep 16 13:32:16.792124 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.791720 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7d5668779d-z7sst"] Sep 16 13:32:16.792124 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.791897 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:16.796983 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.796959 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 16 13:32:16.797281 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.797258 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 16 13:32:16.797465 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.797448 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 16 13:32:16.797647 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.797625 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-t8ghf\"" Sep 16 13:32:16.797818 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.797794 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 16 13:32:16.797927 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.797531 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 16 13:32:16.798144 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.798129 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dn4q1v2b68snp\"" Sep 16 13:32:16.848576 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.848538 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2dcd994e-d1f5-4da8-b4a7-9627a565cef4-secret-grpc-tls\") pod \"thanos-querier-7d5668779d-z7sst\" (UID: \"2dcd994e-d1f5-4da8-b4a7-9627a565cef4\") " pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:16.848793 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.848579 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2dcd994e-d1f5-4da8-b4a7-9627a565cef4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d5668779d-z7sst\" (UID: \"2dcd994e-d1f5-4da8-b4a7-9627a565cef4\") " pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:16.848793 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.848608 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2dcd994e-d1f5-4da8-b4a7-9627a565cef4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d5668779d-z7sst\" (UID: \"2dcd994e-d1f5-4da8-b4a7-9627a565cef4\") " pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:16.848793 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.848647 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4nqn\" (UniqueName: \"kubernetes.io/projected/2dcd994e-d1f5-4da8-b4a7-9627a565cef4-kube-api-access-v4nqn\") pod \"thanos-querier-7d5668779d-z7sst\" (UID: \"2dcd994e-d1f5-4da8-b4a7-9627a565cef4\") " pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:16.848793 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.848680 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2dcd994e-d1f5-4da8-b4a7-9627a565cef4-metrics-client-ca\") pod \"thanos-querier-7d5668779d-z7sst\" (UID: \"2dcd994e-d1f5-4da8-b4a7-9627a565cef4\") " pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:16.848793 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.848706 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2dcd994e-d1f5-4da8-b4a7-9627a565cef4-secret-thanos-querier-tls\") pod \"thanos-querier-7d5668779d-z7sst\" (UID: \"2dcd994e-d1f5-4da8-b4a7-9627a565cef4\") " pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:16.848793 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.848767 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2dcd994e-d1f5-4da8-b4a7-9627a565cef4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d5668779d-z7sst\" (UID: \"2dcd994e-d1f5-4da8-b4a7-9627a565cef4\") " pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:16.849039 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.848806 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2dcd994e-d1f5-4da8-b4a7-9627a565cef4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d5668779d-z7sst\" (UID: \"2dcd994e-d1f5-4da8-b4a7-9627a565cef4\") " pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:16.949810 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.949776 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2dcd994e-d1f5-4da8-b4a7-9627a565cef4-secret-thanos-querier-tls\") pod \"thanos-querier-7d5668779d-z7sst\" (UID: \"2dcd994e-d1f5-4da8-b4a7-9627a565cef4\") " pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:16.949958 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.949845 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2dcd994e-d1f5-4da8-b4a7-9627a565cef4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d5668779d-z7sst\" (UID: \"2dcd994e-d1f5-4da8-b4a7-9627a565cef4\") " pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:16.949958 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.949863 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2dcd994e-d1f5-4da8-b4a7-9627a565cef4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d5668779d-z7sst\" (UID: \"2dcd994e-d1f5-4da8-b4a7-9627a565cef4\") " pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:16.949958 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.949905 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2dcd994e-d1f5-4da8-b4a7-9627a565cef4-secret-grpc-tls\") pod \"thanos-querier-7d5668779d-z7sst\" (UID: \"2dcd994e-d1f5-4da8-b4a7-9627a565cef4\") " pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:16.949958 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.949924 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2dcd994e-d1f5-4da8-b4a7-9627a565cef4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d5668779d-z7sst\" (UID: \"2dcd994e-d1f5-4da8-b4a7-9627a565cef4\") " pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:16.949958 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.949950 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2dcd994e-d1f5-4da8-b4a7-9627a565cef4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d5668779d-z7sst\" (UID: \"2dcd994e-d1f5-4da8-b4a7-9627a565cef4\") " pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:16.950177 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.949985 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4nqn\" (UniqueName: \"kubernetes.io/projected/2dcd994e-d1f5-4da8-b4a7-9627a565cef4-kube-api-access-v4nqn\") pod \"thanos-querier-7d5668779d-z7sst\" (UID: \"2dcd994e-d1f5-4da8-b4a7-9627a565cef4\") " pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:16.950177 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.950015 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2dcd994e-d1f5-4da8-b4a7-9627a565cef4-metrics-client-ca\") pod \"thanos-querier-7d5668779d-z7sst\" (UID: \"2dcd994e-d1f5-4da8-b4a7-9627a565cef4\") " pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:16.950787 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.950756 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2dcd994e-d1f5-4da8-b4a7-9627a565cef4-metrics-client-ca\") pod \"thanos-querier-7d5668779d-z7sst\" (UID: \"2dcd994e-d1f5-4da8-b4a7-9627a565cef4\") " pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:16.955027 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.954638 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2dcd994e-d1f5-4da8-b4a7-9627a565cef4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d5668779d-z7sst\" (UID: \"2dcd994e-d1f5-4da8-b4a7-9627a565cef4\") " pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:16.955027 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.954990 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2dcd994e-d1f5-4da8-b4a7-9627a565cef4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d5668779d-z7sst\" (UID: \"2dcd994e-d1f5-4da8-b4a7-9627a565cef4\") " pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:16.955662 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.955617 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2dcd994e-d1f5-4da8-b4a7-9627a565cef4-secret-thanos-querier-tls\") pod \"thanos-querier-7d5668779d-z7sst\" (UID: \"2dcd994e-d1f5-4da8-b4a7-9627a565cef4\") " pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:16.962264 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.962185 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2dcd994e-d1f5-4da8-b4a7-9627a565cef4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d5668779d-z7sst\" (UID: \"2dcd994e-d1f5-4da8-b4a7-9627a565cef4\") " pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:16.963849 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.963824 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4nqn\" (UniqueName: \"kubernetes.io/projected/2dcd994e-d1f5-4da8-b4a7-9627a565cef4-kube-api-access-v4nqn\") pod \"thanos-querier-7d5668779d-z7sst\" (UID: \"2dcd994e-d1f5-4da8-b4a7-9627a565cef4\") " pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:16.968274 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.968147 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2dcd994e-d1f5-4da8-b4a7-9627a565cef4-secret-grpc-tls\") pod \"thanos-querier-7d5668779d-z7sst\" (UID: \"2dcd994e-d1f5-4da8-b4a7-9627a565cef4\") " pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:16.968684 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:16.968662 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2dcd994e-d1f5-4da8-b4a7-9627a565cef4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d5668779d-z7sst\" (UID: \"2dcd994e-d1f5-4da8-b4a7-9627a565cef4\") " pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:17.109258 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:17.109169 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:17.614881 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:17.614854 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7d5668779d-z7sst"] Sep 16 13:32:17.716104 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:17.716079 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vg66w" Sep 16 13:32:17.728514 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:32:17.728483 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dcd994e_d1f5_4da8_b4a7_9627a565cef4.slice/crio-0de3319625c789d6fe4e15be7620c3fbc4490de9a653c8e190df393f5c083321 WatchSource:0}: Error finding container 0de3319625c789d6fe4e15be7620c3fbc4490de9a653c8e190df393f5c083321: Status 404 returned error can't find the container with id 0de3319625c789d6fe4e15be7620c3fbc4490de9a653c8e190df393f5c083321 Sep 16 13:32:17.869653 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:17.869622 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" event={"ID":"2dcd994e-d1f5-4da8-b4a7-9627a565cef4","Type":"ContainerStarted","Data":"0de3319625c789d6fe4e15be7620c3fbc4490de9a653c8e190df393f5c083321"} Sep 16 13:32:18.876180 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:18.876145 2562 generic.go:356] "Generic (PLEG): container finished" podID="624e58c2-824b-4b27-8f19-56f047656011" containerID="74de19aa30e7f9616176afc675b39b6eb8c8485064b91691442ac8e1871ff9a4" exitCode=0 Sep 16 13:32:18.876609 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:18.876244 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jttjb" event={"ID":"624e58c2-824b-4b27-8f19-56f047656011","Type":"ContainerDied","Data":"74de19aa30e7f9616176afc675b39b6eb8c8485064b91691442ac8e1871ff9a4"} Sep 16 13:32:19.168590 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.167294 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-84cd89899f-vvm5g"] Sep 16 13:32:19.185950 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.184528 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84cd89899f-vvm5g"] Sep 16 13:32:19.185950 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.184641 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:19.191218 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.190848 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 16 13:32:19.191218 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.190904 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-vg24qh5mscd3\"" Sep 16 13:32:19.191218 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.191145 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 16 13:32:19.191473 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.191334 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 16 13:32:19.191928 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.191561 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 16 13:32:19.191928 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.191784 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-sczrc\"" Sep 16 13:32:19.271965 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.271928 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rffcb\" (UniqueName: \"kubernetes.io/projected/44cef9df-a46b-44ab-8e3e-c3552ebc9174-kube-api-access-rffcb\") pod \"metrics-server-84cd89899f-vvm5g\" (UID: \"44cef9df-a46b-44ab-8e3e-c3552ebc9174\") " pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:19.271965 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.271984 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/44cef9df-a46b-44ab-8e3e-c3552ebc9174-secret-metrics-server-client-certs\") pod \"metrics-server-84cd89899f-vvm5g\" (UID: \"44cef9df-a46b-44ab-8e3e-c3552ebc9174\") " pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:19.272219 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.272066 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/44cef9df-a46b-44ab-8e3e-c3552ebc9174-metrics-server-audit-profiles\") pod \"metrics-server-84cd89899f-vvm5g\" (UID: \"44cef9df-a46b-44ab-8e3e-c3552ebc9174\") " pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:19.272219 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.272111 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/44cef9df-a46b-44ab-8e3e-c3552ebc9174-audit-log\") pod \"metrics-server-84cd89899f-vvm5g\" (UID: \"44cef9df-a46b-44ab-8e3e-c3552ebc9174\") " pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:19.272219 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.272165 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/44cef9df-a46b-44ab-8e3e-c3552ebc9174-secret-metrics-server-tls\") pod \"metrics-server-84cd89899f-vvm5g\" (UID: \"44cef9df-a46b-44ab-8e3e-c3552ebc9174\") " pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:19.272219 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.272190 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44cef9df-a46b-44ab-8e3e-c3552ebc9174-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84cd89899f-vvm5g\" (UID: \"44cef9df-a46b-44ab-8e3e-c3552ebc9174\") " pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:19.272436 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.272266 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/44cef9df-a46b-44ab-8e3e-c3552ebc9174-client-ca-bundle\") pod \"metrics-server-84cd89899f-vvm5g\" (UID: \"44cef9df-a46b-44ab-8e3e-c3552ebc9174\") " pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:19.373746 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.373711 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/44cef9df-a46b-44ab-8e3e-c3552ebc9174-secret-metrics-server-client-certs\") pod \"metrics-server-84cd89899f-vvm5g\" (UID: \"44cef9df-a46b-44ab-8e3e-c3552ebc9174\") " pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:19.375277 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.373775 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/44cef9df-a46b-44ab-8e3e-c3552ebc9174-metrics-server-audit-profiles\") pod \"metrics-server-84cd89899f-vvm5g\" (UID: \"44cef9df-a46b-44ab-8e3e-c3552ebc9174\") " pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:19.375277 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.373818 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/44cef9df-a46b-44ab-8e3e-c3552ebc9174-audit-log\") pod \"metrics-server-84cd89899f-vvm5g\" (UID: \"44cef9df-a46b-44ab-8e3e-c3552ebc9174\") " pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:19.375277 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.373879 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/44cef9df-a46b-44ab-8e3e-c3552ebc9174-secret-metrics-server-tls\") pod \"metrics-server-84cd89899f-vvm5g\" (UID: \"44cef9df-a46b-44ab-8e3e-c3552ebc9174\") " pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:19.375277 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.373916 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44cef9df-a46b-44ab-8e3e-c3552ebc9174-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84cd89899f-vvm5g\" (UID: \"44cef9df-a46b-44ab-8e3e-c3552ebc9174\") " pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:19.375277 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.373955 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/44cef9df-a46b-44ab-8e3e-c3552ebc9174-client-ca-bundle\") pod \"metrics-server-84cd89899f-vvm5g\" (UID: \"44cef9df-a46b-44ab-8e3e-c3552ebc9174\") " pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:19.375277 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.373989 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rffcb\" (UniqueName: \"kubernetes.io/projected/44cef9df-a46b-44ab-8e3e-c3552ebc9174-kube-api-access-rffcb\") pod \"metrics-server-84cd89899f-vvm5g\" (UID: \"44cef9df-a46b-44ab-8e3e-c3552ebc9174\") " pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:19.375277 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.375174 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44cef9df-a46b-44ab-8e3e-c3552ebc9174-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84cd89899f-vvm5g\" (UID: \"44cef9df-a46b-44ab-8e3e-c3552ebc9174\") " pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:19.375652 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.375459 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/44cef9df-a46b-44ab-8e3e-c3552ebc9174-metrics-server-audit-profiles\") pod \"metrics-server-84cd89899f-vvm5g\" (UID: \"44cef9df-a46b-44ab-8e3e-c3552ebc9174\") " pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:19.375747 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.375720 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/44cef9df-a46b-44ab-8e3e-c3552ebc9174-audit-log\") pod \"metrics-server-84cd89899f-vvm5g\" (UID: \"44cef9df-a46b-44ab-8e3e-c3552ebc9174\") " pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:19.377840 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.377405 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/44cef9df-a46b-44ab-8e3e-c3552ebc9174-client-ca-bundle\") pod \"metrics-server-84cd89899f-vvm5g\" (UID: \"44cef9df-a46b-44ab-8e3e-c3552ebc9174\") " pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:19.377840 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.377812 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/44cef9df-a46b-44ab-8e3e-c3552ebc9174-secret-metrics-server-tls\") pod \"metrics-server-84cd89899f-vvm5g\" (UID: \"44cef9df-a46b-44ab-8e3e-c3552ebc9174\") " pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:19.379155 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.379134 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/44cef9df-a46b-44ab-8e3e-c3552ebc9174-secret-metrics-server-client-certs\") pod \"metrics-server-84cd89899f-vvm5g\" (UID: \"44cef9df-a46b-44ab-8e3e-c3552ebc9174\") " pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:19.384200 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.384163 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rffcb\" (UniqueName: \"kubernetes.io/projected/44cef9df-a46b-44ab-8e3e-c3552ebc9174-kube-api-access-rffcb\") pod \"metrics-server-84cd89899f-vvm5g\" (UID: \"44cef9df-a46b-44ab-8e3e-c3552ebc9174\") " pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:19.503080 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.502803 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:19.558402 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.558352 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-t84x2"] Sep 16 13:32:19.584802 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.582013 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-t84x2"] Sep 16 13:32:19.584802 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.582122 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t84x2" Sep 16 13:32:19.586607 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.585839 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-zfb6l\"" Sep 16 13:32:19.586607 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.586060 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 16 13:32:19.680350 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.680315 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtk27\" (UniqueName: \"kubernetes.io/projected/cab7002e-0e52-4417-84c1-c00e5da634b7-kube-api-access-jtk27\") pod \"monitoring-plugin-5d85778b9d-t84x2\" (UID: \"cab7002e-0e52-4417-84c1-c00e5da634b7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t84x2" Sep 16 13:32:19.680539 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.680361 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cab7002e-0e52-4417-84c1-c00e5da634b7-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-t84x2\" (UID: \"cab7002e-0e52-4417-84c1-c00e5da634b7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t84x2" Sep 16 13:32:19.781264 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.781156 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtk27\" (UniqueName: \"kubernetes.io/projected/cab7002e-0e52-4417-84c1-c00e5da634b7-kube-api-access-jtk27\") pod \"monitoring-plugin-5d85778b9d-t84x2\" (UID: \"cab7002e-0e52-4417-84c1-c00e5da634b7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t84x2" Sep 16 13:32:19.781264 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.781234 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cab7002e-0e52-4417-84c1-c00e5da634b7-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-t84x2\" (UID: \"cab7002e-0e52-4417-84c1-c00e5da634b7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t84x2" Sep 16 13:32:19.781554 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:32:19.781447 2562 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 16 13:32:19.781554 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:32:19.781513 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cab7002e-0e52-4417-84c1-c00e5da634b7-monitoring-plugin-cert podName:cab7002e-0e52-4417-84c1-c00e5da634b7 nodeName:}" failed. No retries permitted until 2026-09-16 13:32:20.28149033 +0000 UTC m=+101.325793279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/cab7002e-0e52-4417-84c1-c00e5da634b7-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-t84x2" (UID: "cab7002e-0e52-4417-84c1-c00e5da634b7") : secret "monitoring-plugin-cert" not found Sep 16 13:32:19.791897 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.791847 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtk27\" (UniqueName: \"kubernetes.io/projected/cab7002e-0e52-4417-84c1-c00e5da634b7-kube-api-access-jtk27\") pod \"monitoring-plugin-5d85778b9d-t84x2\" (UID: \"cab7002e-0e52-4417-84c1-c00e5da634b7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t84x2" Sep 16 13:32:19.848507 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.847413 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84cd89899f-vvm5g"] Sep 16 13:32:19.852195 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:32:19.852039 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44cef9df_a46b_44ab_8e3e_c3552ebc9174.slice/crio-9e77e709479b96933cc8ca61a1629ced9cb64dd6c0c5bfbbd62cdf22436181a9 WatchSource:0}: Error finding container 9e77e709479b96933cc8ca61a1629ced9cb64dd6c0c5bfbbd62cdf22436181a9: Status 404 returned error can't find the container with id 9e77e709479b96933cc8ca61a1629ced9cb64dd6c0c5bfbbd62cdf22436181a9 Sep 16 13:32:19.885121 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.885052 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" event={"ID":"44cef9df-a46b-44ab-8e3e-c3552ebc9174","Type":"ContainerStarted","Data":"9e77e709479b96933cc8ca61a1629ced9cb64dd6c0c5bfbbd62cdf22436181a9"} Sep 16 13:32:19.888135 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.888107 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jttjb" event={"ID":"624e58c2-824b-4b27-8f19-56f047656011","Type":"ContainerStarted","Data":"0a417c3f95e7538f02f970f71830691a4a8e2fe89cc7987c36cc17720d80a0d3"} Sep 16 13:32:19.888255 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.888160 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jttjb" event={"ID":"624e58c2-824b-4b27-8f19-56f047656011","Type":"ContainerStarted","Data":"17bde466f9efb3cec301ce6c3484b89636abac08dab3d750a91548ace8cd4d8e"} Sep 16 13:32:19.912325 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:19.912282 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jttjb" podStartSLOduration=3.917976144 podStartE2EDuration="5.91226796s" totalImagesPullingTime="1.900608818s" totalInitContainerRuntime="93.682998ms" isStatefulPod=true podCreationTimestamp="2026-09-16 13:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:32:19.910450948 +0000 UTC m=+100.954753935" watchObservedRunningTime="2026-09-16 13:32:19.91226796 +0000 UTC m=+100.956570925" Sep 16 13:32:20.287099 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:20.287060 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cab7002e-0e52-4417-84c1-c00e5da634b7-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-t84x2\" (UID: \"cab7002e-0e52-4417-84c1-c00e5da634b7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t84x2" Sep 16 13:32:20.290280 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:20.290252 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cab7002e-0e52-4417-84c1-c00e5da634b7-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-t84x2\" (UID: \"cab7002e-0e52-4417-84c1-c00e5da634b7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t84x2" Sep 16 13:32:20.497680 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:20.497648 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t84x2" Sep 16 13:32:21.089542 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:21.089513 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-t84x2"] Sep 16 13:32:21.219563 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:32:21.219527 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab7002e_0e52_4417_84c1_c00e5da634b7.slice/crio-25be85cbfd8cd4855cb3978cedc87409fe51ebeb62ab245991d2dda12c54da91 WatchSource:0}: Error finding container 25be85cbfd8cd4855cb3978cedc87409fe51ebeb62ab245991d2dda12c54da91: Status 404 returned error can't find the container with id 25be85cbfd8cd4855cb3978cedc87409fe51ebeb62ab245991d2dda12c54da91 Sep 16 13:32:21.918456 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:21.918422 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" event={"ID":"2dcd994e-d1f5-4da8-b4a7-9627a565cef4","Type":"ContainerStarted","Data":"f61fb478d059de5d2ebcd9c778606163e1f2d8f9f77696c39c0e88529f2e2753"} Sep 16 13:32:21.918672 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:21.918488 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" event={"ID":"2dcd994e-d1f5-4da8-b4a7-9627a565cef4","Type":"ContainerStarted","Data":"2c4d65c4fe16b7a4619d3225f2655b6375d61e69f15883c6970183ffa38c1866"} Sep 16 13:32:21.918672 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:21.918517 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" event={"ID":"2dcd994e-d1f5-4da8-b4a7-9627a565cef4","Type":"ContainerStarted","Data":"e41cf0c6de5172dc71a78239868ef994aa0a9dc7eac37b53318117b07269892f"} Sep 16 13:32:21.919490 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:21.919465 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t84x2" event={"ID":"cab7002e-0e52-4417-84c1-c00e5da634b7","Type":"ContainerStarted","Data":"25be85cbfd8cd4855cb3978cedc87409fe51ebeb62ab245991d2dda12c54da91"} Sep 16 13:32:23.929770 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:23.929737 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" event={"ID":"44cef9df-a46b-44ab-8e3e-c3552ebc9174","Type":"ContainerStarted","Data":"5dffbe1d84fa24c92e9fb48c57897d952e4a06595d81717ebf92b28678957f8d"} Sep 16 13:32:23.932249 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:23.932220 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" event={"ID":"2dcd994e-d1f5-4da8-b4a7-9627a565cef4","Type":"ContainerStarted","Data":"7e7015c89c76dec20cf53ebfc856a58f0c2103a09da021616f8bec5ea10f2426"} Sep 16 13:32:23.933762 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:23.933736 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t84x2" event={"ID":"cab7002e-0e52-4417-84c1-c00e5da634b7","Type":"ContainerStarted","Data":"e80829bca0bd2f90fb2935a3c545977c77185941ae82c088a2c30c639ba1f4ba"} Sep 16 13:32:23.938621 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:23.938565 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t84x2" Sep 16 13:32:23.943394 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:23.943344 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t84x2" Sep 16 13:32:23.955961 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:23.955924 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-t84x2" podStartSLOduration=2.451463654 podStartE2EDuration="4.955909609s" totalImagesPullingTime="2.504445955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:32:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:32:23.955492456 +0000 UTC m=+104.999795423" watchObservedRunningTime="2026-09-16 13:32:23.955909609 +0000 UTC m=+105.000212588" Sep 16 13:32:23.981434 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:23.978117 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" podStartSLOduration=1.125610742 podStartE2EDuration="4.978100103s" totalImagesPullingTime="3.852489361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:32:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:32:23.974684111 +0000 UTC m=+105.018987098" watchObservedRunningTime="2026-09-16 13:32:23.978100103 +0000 UTC m=+105.022403070" Sep 16 13:32:24.940361 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:24.940316 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" event={"ID":"2dcd994e-d1f5-4da8-b4a7-9627a565cef4","Type":"ContainerStarted","Data":"4461805cb4672e628a797fa785e2d6a01fcbcdb75683051fd000ab72c94ba787"} Sep 16 13:32:24.940788 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:24.940457 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" event={"ID":"2dcd994e-d1f5-4da8-b4a7-9627a565cef4","Type":"ContainerStarted","Data":"19f8a44778efc9d344dbc945ae9b5420919e9e0025250b3cbb9bc7aec57f8bfd"} Sep 16 13:32:24.940788 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:24.940713 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:24.970047 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:24.970002 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" podStartSLOduration=3.3632067709999998 podStartE2EDuration="8.969988789s" totalImagesPullingTime="5.606782018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:32:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:32:24.967520739 +0000 UTC m=+106.011823725" watchObservedRunningTime="2026-09-16 13:32:24.969988789 +0000 UTC m=+106.014291755" Sep 16 13:32:30.951613 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:30.951585 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7d5668779d-z7sst" Sep 16 13:32:37.999991 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:37.999957 2562 generic.go:356] "Generic (PLEG): container finished" podID="8618820e-6a4d-461e-bf89-c84a72b217fd" containerID="06d5a50a051b7e6f420feebe102ef7c6011b590031f36b5f6681a4b6f479bd83" exitCode=0 Sep 16 13:32:38.000457 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:38.000038 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hnlgr" event={"ID":"8618820e-6a4d-461e-bf89-c84a72b217fd","Type":"ContainerDied","Data":"06d5a50a051b7e6f420feebe102ef7c6011b590031f36b5f6681a4b6f479bd83"} Sep 16 13:32:38.000457 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:38.000398 2562 scope.go:117] "RemoveContainer" containerID="06d5a50a051b7e6f420feebe102ef7c6011b590031f36b5f6681a4b6f479bd83" Sep 16 13:32:39.006208 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:39.006176 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hnlgr" event={"ID":"8618820e-6a4d-461e-bf89-c84a72b217fd","Type":"ContainerStarted","Data":"4b3e68d593ac0010884fc9b482dc4f3a53a310355ec46cfbc18d74a981619491"} Sep 16 13:32:39.504204 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:39.504170 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:39.504412 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:39.504244 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:41.369290 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.369235 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" podUID="58c0f9ae-bb5d-4621-b804-29b5a3a88f6c" containerName="registry" containerID="cri-o://d6cbc34decf12e65eaaff6b44bc1eb675f628b30e6127b45cccbeddce2dfdb85" gracePeriod=30 Sep 16 13:32:41.643199 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.643144 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:32:41.761803 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.761772 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-tls\") pod \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " Sep 16 13:32:41.761956 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.761851 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-image-registry-private-configuration\") pod \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " Sep 16 13:32:41.761956 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.761888 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-ca-trust-extracted\") pod \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " Sep 16 13:32:41.761956 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.761919 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-bound-sa-token\") pod \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " Sep 16 13:32:41.761956 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.761945 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw5b5\" (UniqueName: \"kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-kube-api-access-hw5b5\") pod \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " Sep 16 13:32:41.762152 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.761959 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-certificates\") pod \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " Sep 16 13:32:41.762152 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.762000 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-installation-pull-secrets\") pod \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " Sep 16 13:32:41.762152 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.762014 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-trusted-ca\") pod \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\" (UID: \"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c\") " Sep 16 13:32:41.762666 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.762585 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "58c0f9ae-bb5d-4621-b804-29b5a3a88f6c" (UID: "58c0f9ae-bb5d-4621-b804-29b5a3a88f6c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:32:41.762881 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.762791 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "58c0f9ae-bb5d-4621-b804-29b5a3a88f6c" (UID: "58c0f9ae-bb5d-4621-b804-29b5a3a88f6c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:32:41.764573 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.764523 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "58c0f9ae-bb5d-4621-b804-29b5a3a88f6c" (UID: "58c0f9ae-bb5d-4621-b804-29b5a3a88f6c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:32:41.764690 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.764603 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-kube-api-access-hw5b5" (OuterVolumeSpecName: "kube-api-access-hw5b5") pod "58c0f9ae-bb5d-4621-b804-29b5a3a88f6c" (UID: "58c0f9ae-bb5d-4621-b804-29b5a3a88f6c"). InnerVolumeSpecName "kube-api-access-hw5b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:32:41.764805 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.764689 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "58c0f9ae-bb5d-4621-b804-29b5a3a88f6c" (UID: "58c0f9ae-bb5d-4621-b804-29b5a3a88f6c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:32:41.764978 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.764892 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "58c0f9ae-bb5d-4621-b804-29b5a3a88f6c" (UID: "58c0f9ae-bb5d-4621-b804-29b5a3a88f6c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:32:41.765024 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.764983 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "58c0f9ae-bb5d-4621-b804-29b5a3a88f6c" (UID: "58c0f9ae-bb5d-4621-b804-29b5a3a88f6c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:32:41.770772 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.770751 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "58c0f9ae-bb5d-4621-b804-29b5a3a88f6c" (UID: "58c0f9ae-bb5d-4621-b804-29b5a3a88f6c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:32:41.862991 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.862958 2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-bound-sa-token\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:32:41.862991 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.862986 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw5b5\" (UniqueName: \"kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-kube-api-access-hw5b5\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:32:41.863156 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.863000 2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-certificates\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:32:41.863156 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.863012 2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-installation-pull-secrets\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:32:41.863156 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.863023 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-trusted-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:32:41.863156 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.863033 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-registry-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:32:41.863156 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.863045 2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-image-registry-private-configuration\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:32:41.863156 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:41.863057 2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c-ca-trust-extracted\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:32:42.019411 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:42.019363 2562 generic.go:356] "Generic (PLEG): container finished" podID="58c0f9ae-bb5d-4621-b804-29b5a3a88f6c" containerID="d6cbc34decf12e65eaaff6b44bc1eb675f628b30e6127b45cccbeddce2dfdb85" exitCode=0 Sep 16 13:32:42.019581 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:42.019424 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" event={"ID":"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c","Type":"ContainerDied","Data":"d6cbc34decf12e65eaaff6b44bc1eb675f628b30e6127b45cccbeddce2dfdb85"} Sep 16 13:32:42.019581 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:42.019432 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" Sep 16 13:32:42.019581 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:42.019463 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6449dbdc57-jc64r" event={"ID":"58c0f9ae-bb5d-4621-b804-29b5a3a88f6c","Type":"ContainerDied","Data":"e95d5e468f7201f5941c578b017ba4bcf9ebdb6e985988b65a395dbadbfe80b8"} Sep 16 13:32:42.019581 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:42.019482 2562 scope.go:117] "RemoveContainer" containerID="d6cbc34decf12e65eaaff6b44bc1eb675f628b30e6127b45cccbeddce2dfdb85" Sep 16 13:32:42.028475 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:42.028459 2562 scope.go:117] "RemoveContainer" containerID="d6cbc34decf12e65eaaff6b44bc1eb675f628b30e6127b45cccbeddce2dfdb85" Sep 16 13:32:42.033438 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:32:42.028969 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6cbc34decf12e65eaaff6b44bc1eb675f628b30e6127b45cccbeddce2dfdb85\": container with ID starting with d6cbc34decf12e65eaaff6b44bc1eb675f628b30e6127b45cccbeddce2dfdb85 not found: ID does not exist" containerID="d6cbc34decf12e65eaaff6b44bc1eb675f628b30e6127b45cccbeddce2dfdb85" Sep 16 13:32:42.033438 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:42.029022 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6cbc34decf12e65eaaff6b44bc1eb675f628b30e6127b45cccbeddce2dfdb85"} err="failed to get container status \"d6cbc34decf12e65eaaff6b44bc1eb675f628b30e6127b45cccbeddce2dfdb85\": rpc error: code = NotFound desc = could not find container \"d6cbc34decf12e65eaaff6b44bc1eb675f628b30e6127b45cccbeddce2dfdb85\": container with ID starting with d6cbc34decf12e65eaaff6b44bc1eb675f628b30e6127b45cccbeddce2dfdb85 not found: ID does not exist" Sep 16 13:32:42.042808 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:42.042788 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6449dbdc57-jc64r"] Sep 16 13:32:42.046880 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:42.046861 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6449dbdc57-jc64r"] Sep 16 13:32:43.412321 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:43.412290 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58c0f9ae-bb5d-4621-b804-29b5a3a88f6c" path="/var/lib/kubelet/pods/58c0f9ae-bb5d-4621-b804-29b5a3a88f6c/volumes" Sep 16 13:32:48.045212 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:48.045132 2562 generic.go:356] "Generic (PLEG): container finished" podID="1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b" containerID="c388f5c713f8187c9a42ce29e84759ace04823829b44f74b1d8e77d2ec200a12" exitCode=0 Sep 16 13:32:48.045212 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:48.045183 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-zb868" event={"ID":"1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b","Type":"ContainerDied","Data":"c388f5c713f8187c9a42ce29e84759ace04823829b44f74b1d8e77d2ec200a12"} Sep 16 13:32:48.045634 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:48.045494 2562 scope.go:117] "RemoveContainer" containerID="c388f5c713f8187c9a42ce29e84759ace04823829b44f74b1d8e77d2ec200a12" Sep 16 13:32:49.050824 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:49.050792 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-zb868" event={"ID":"1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b","Type":"ContainerStarted","Data":"f6df62f3f37103cf67da5995acb52c2d9185f22a2a080eed619266c56500c3bf"} Sep 16 13:32:58.090882 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:58.090810 2562 generic.go:356] "Generic (PLEG): container finished" podID="647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3" containerID="0bf800124a0716c7dc41382bd93c24610b893571029483f307096b3bb844b3e1" exitCode=0 Sep 16 13:32:58.091216 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:58.090884 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-vdgqh" event={"ID":"647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3","Type":"ContainerDied","Data":"0bf800124a0716c7dc41382bd93c24610b893571029483f307096b3bb844b3e1"} Sep 16 13:32:58.091216 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:58.091192 2562 scope.go:117] "RemoveContainer" containerID="0bf800124a0716c7dc41382bd93c24610b893571029483f307096b3bb844b3e1" Sep 16 13:32:59.096820 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:59.096782 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-vdgqh" event={"ID":"647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3","Type":"ContainerStarted","Data":"b2f565cfaefc572d656ef3763eff1c656537ecd1193338a8a161aae31636cf52"} Sep 16 13:32:59.509359 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:59.509332 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:32:59.513076 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:32:59.513052 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-84cd89899f-vvm5g" Sep 16 13:35:18.280276 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:18.280243 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn"] Sep 16 13:35:18.280798 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:18.280673 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58c0f9ae-bb5d-4621-b804-29b5a3a88f6c" containerName="registry" Sep 16 13:35:18.280798 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:18.280690 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="58c0f9ae-bb5d-4621-b804-29b5a3a88f6c" containerName="registry" Sep 16 13:35:18.280798 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:18.280791 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="58c0f9ae-bb5d-4621-b804-29b5a3a88f6c" containerName="registry" Sep 16 13:35:18.282859 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:18.282841 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn" Sep 16 13:35:18.285578 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:18.285558 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 13:35:18.285682 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:18.285558 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-75dbt\"" Sep 16 13:35:18.285682 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:18.285559 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 13:35:18.294528 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:18.294511 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn"] Sep 16 13:35:18.336410 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:18.336366 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7vvp\" (UniqueName: \"kubernetes.io/projected/9a7ee637-e41f-4a06-b95a-81828cfce3c2-kube-api-access-p7vvp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn\" (UID: \"9a7ee637-e41f-4a06-b95a-81828cfce3c2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn" Sep 16 13:35:18.336504 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:18.336437 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a7ee637-e41f-4a06-b95a-81828cfce3c2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn\" (UID: \"9a7ee637-e41f-4a06-b95a-81828cfce3c2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn" Sep 16 13:35:18.336504 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:18.336468 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a7ee637-e41f-4a06-b95a-81828cfce3c2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn\" (UID: \"9a7ee637-e41f-4a06-b95a-81828cfce3c2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn" Sep 16 13:35:18.437496 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:18.437468 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a7ee637-e41f-4a06-b95a-81828cfce3c2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn\" (UID: \"9a7ee637-e41f-4a06-b95a-81828cfce3c2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn" Sep 16 13:35:18.437637 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:18.437525 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7vvp\" (UniqueName: \"kubernetes.io/projected/9a7ee637-e41f-4a06-b95a-81828cfce3c2-kube-api-access-p7vvp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn\" (UID: \"9a7ee637-e41f-4a06-b95a-81828cfce3c2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn" Sep 16 13:35:18.437637 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:18.437556 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a7ee637-e41f-4a06-b95a-81828cfce3c2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn\" (UID: \"9a7ee637-e41f-4a06-b95a-81828cfce3c2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn" Sep 16 13:35:18.437887 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:18.437864 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a7ee637-e41f-4a06-b95a-81828cfce3c2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn\" (UID: \"9a7ee637-e41f-4a06-b95a-81828cfce3c2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn" Sep 16 13:35:18.437946 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:18.437877 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a7ee637-e41f-4a06-b95a-81828cfce3c2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn\" (UID: \"9a7ee637-e41f-4a06-b95a-81828cfce3c2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn" Sep 16 13:35:18.446238 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:18.446210 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7vvp\" (UniqueName: \"kubernetes.io/projected/9a7ee637-e41f-4a06-b95a-81828cfce3c2-kube-api-access-p7vvp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn\" (UID: \"9a7ee637-e41f-4a06-b95a-81828cfce3c2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn" Sep 16 13:35:18.591691 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:18.591626 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn" Sep 16 13:35:18.862925 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:18.862898 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn"] Sep 16 13:35:18.865433 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:35:18.865407 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7ee637_e41f_4a06_b95a_81828cfce3c2.slice/crio-be93d9515b61c75a2fdaeea7012c79bef07b94588084bab45b3aa83c77005a46 WatchSource:0}: Error finding container be93d9515b61c75a2fdaeea7012c79bef07b94588084bab45b3aa83c77005a46: Status 404 returned error can't find the container with id be93d9515b61c75a2fdaeea7012c79bef07b94588084bab45b3aa83c77005a46 Sep 16 13:35:19.634423 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:19.634370 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn" event={"ID":"9a7ee637-e41f-4a06-b95a-81828cfce3c2","Type":"ContainerStarted","Data":"be93d9515b61c75a2fdaeea7012c79bef07b94588084bab45b3aa83c77005a46"} Sep 16 13:35:25.664099 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:25.664069 2562 generic.go:356] "Generic (PLEG): container finished" podID="9a7ee637-e41f-4a06-b95a-81828cfce3c2" containerID="b9c9ff6d0c74c0c10f313effd1298686d9e287362f00c178282b6a9c9ed3a3f4" exitCode=0 Sep 16 13:35:25.664502 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:25.664157 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn" event={"ID":"9a7ee637-e41f-4a06-b95a-81828cfce3c2","Type":"ContainerDied","Data":"b9c9ff6d0c74c0c10f313effd1298686d9e287362f00c178282b6a9c9ed3a3f4"} Sep 16 13:35:28.678816 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:28.678783 2562 generic.go:356] "Generic (PLEG): container finished" podID="9a7ee637-e41f-4a06-b95a-81828cfce3c2" containerID="43624f576a1f3c18a38a5511be29946f8502d92aa105d210933ad9a22fbf4625" exitCode=0 Sep 16 13:35:28.679190 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:28.678869 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn" event={"ID":"9a7ee637-e41f-4a06-b95a-81828cfce3c2","Type":"ContainerDied","Data":"43624f576a1f3c18a38a5511be29946f8502d92aa105d210933ad9a22fbf4625"} Sep 16 13:35:35.715071 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:35.715031 2562 generic.go:356] "Generic (PLEG): container finished" podID="9a7ee637-e41f-4a06-b95a-81828cfce3c2" containerID="3d2023540c7e89eb0fa1c417a9c620f3a627c9c82d27b3bfdb360738415354e5" exitCode=0 Sep 16 13:35:35.715071 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:35.715073 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn" event={"ID":"9a7ee637-e41f-4a06-b95a-81828cfce3c2","Type":"ContainerDied","Data":"3d2023540c7e89eb0fa1c417a9c620f3a627c9c82d27b3bfdb360738415354e5"} Sep 16 13:35:36.839280 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:36.839259 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn" Sep 16 13:35:37.000833 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:37.000773 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a7ee637-e41f-4a06-b95a-81828cfce3c2-bundle\") pod \"9a7ee637-e41f-4a06-b95a-81828cfce3c2\" (UID: \"9a7ee637-e41f-4a06-b95a-81828cfce3c2\") " Sep 16 13:35:37.000833 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:37.000801 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a7ee637-e41f-4a06-b95a-81828cfce3c2-util\") pod \"9a7ee637-e41f-4a06-b95a-81828cfce3c2\" (UID: \"9a7ee637-e41f-4a06-b95a-81828cfce3c2\") " Sep 16 13:35:37.000833 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:37.000821 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7vvp\" (UniqueName: \"kubernetes.io/projected/9a7ee637-e41f-4a06-b95a-81828cfce3c2-kube-api-access-p7vvp\") pod \"9a7ee637-e41f-4a06-b95a-81828cfce3c2\" (UID: \"9a7ee637-e41f-4a06-b95a-81828cfce3c2\") " Sep 16 13:35:37.003575 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:37.001815 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7ee637-e41f-4a06-b95a-81828cfce3c2-bundle" (OuterVolumeSpecName: "bundle") pod "9a7ee637-e41f-4a06-b95a-81828cfce3c2" (UID: "9a7ee637-e41f-4a06-b95a-81828cfce3c2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:35:37.004766 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:37.003995 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7ee637-e41f-4a06-b95a-81828cfce3c2-kube-api-access-p7vvp" (OuterVolumeSpecName: "kube-api-access-p7vvp") pod "9a7ee637-e41f-4a06-b95a-81828cfce3c2" (UID: "9a7ee637-e41f-4a06-b95a-81828cfce3c2"). InnerVolumeSpecName "kube-api-access-p7vvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:35:37.007184 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:37.007158 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7ee637-e41f-4a06-b95a-81828cfce3c2-util" (OuterVolumeSpecName: "util") pod "9a7ee637-e41f-4a06-b95a-81828cfce3c2" (UID: "9a7ee637-e41f-4a06-b95a-81828cfce3c2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:35:37.101659 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:37.101639 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a7ee637-e41f-4a06-b95a-81828cfce3c2-bundle\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:35:37.101659 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:37.101660 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a7ee637-e41f-4a06-b95a-81828cfce3c2-util\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:35:37.101768 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:37.101669 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7vvp\" (UniqueName: \"kubernetes.io/projected/9a7ee637-e41f-4a06-b95a-81828cfce3c2-kube-api-access-p7vvp\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:35:37.725002 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:37.724934 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn" Sep 16 13:35:37.725133 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:37.724932 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hb5vn" event={"ID":"9a7ee637-e41f-4a06-b95a-81828cfce3c2","Type":"ContainerDied","Data":"be93d9515b61c75a2fdaeea7012c79bef07b94588084bab45b3aa83c77005a46"} Sep 16 13:35:37.725133 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:37.725045 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be93d9515b61c75a2fdaeea7012c79bef07b94588084bab45b3aa83c77005a46" Sep 16 13:35:39.348747 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:39.348720 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4xmxv_c7828efb-6be7-4f7b-986f-645a2370163a/console-operator/1.log" Sep 16 13:35:39.349240 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:39.349221 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4xmxv_c7828efb-6be7-4f7b-986f-645a2370163a/console-operator/1.log" Sep 16 13:35:39.361205 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:39.361183 2562 kubelet.go:1648] "Image garbage collection succeeded" Sep 16 13:35:43.794248 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:43.794214 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wgx9q"] Sep 16 13:35:43.845711 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:43.794514 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a7ee637-e41f-4a06-b95a-81828cfce3c2" containerName="pull" Sep 16 13:35:43.845711 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:43.794525 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a7ee637-e41f-4a06-b95a-81828cfce3c2" containerName="pull" Sep 16 13:35:43.845711 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:43.794533 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a7ee637-e41f-4a06-b95a-81828cfce3c2" containerName="extract" Sep 16 13:35:43.845711 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:43.794538 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a7ee637-e41f-4a06-b95a-81828cfce3c2" containerName="extract" Sep 16 13:35:43.845711 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:43.794551 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a7ee637-e41f-4a06-b95a-81828cfce3c2" containerName="util" Sep 16 13:35:43.845711 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:43.794556 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a7ee637-e41f-4a06-b95a-81828cfce3c2" containerName="util" Sep 16 13:35:43.845711 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:43.794610 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a7ee637-e41f-4a06-b95a-81828cfce3c2" containerName="extract" Sep 16 13:35:43.854935 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:43.854912 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wgx9q"] Sep 16 13:35:43.855048 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:43.854988 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wgx9q" Sep 16 13:35:43.859862 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:43.859835 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:35:43.859967 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:43.859862 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 16 13:35:43.860986 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:43.860966 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-8cngc\"" Sep 16 13:35:44.050481 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:44.050399 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lppt9\" (UniqueName: \"kubernetes.io/projected/1fdd1cba-7b81-4639-86f5-a8b1802a4fd9-kube-api-access-lppt9\") pod \"cert-manager-operator-controller-manager-55c444bf86-wgx9q\" (UID: \"1fdd1cba-7b81-4639-86f5-a8b1802a4fd9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wgx9q" Sep 16 13:35:44.050481 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:44.050445 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fdd1cba-7b81-4639-86f5-a8b1802a4fd9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wgx9q\" (UID: \"1fdd1cba-7b81-4639-86f5-a8b1802a4fd9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wgx9q" Sep 16 13:35:44.151281 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:44.151254 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fdd1cba-7b81-4639-86f5-a8b1802a4fd9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wgx9q\" (UID: \"1fdd1cba-7b81-4639-86f5-a8b1802a4fd9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wgx9q" Sep 16 13:35:44.151437 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:44.151314 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lppt9\" (UniqueName: \"kubernetes.io/projected/1fdd1cba-7b81-4639-86f5-a8b1802a4fd9-kube-api-access-lppt9\") pod \"cert-manager-operator-controller-manager-55c444bf86-wgx9q\" (UID: \"1fdd1cba-7b81-4639-86f5-a8b1802a4fd9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wgx9q" Sep 16 13:35:44.151656 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:44.151636 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fdd1cba-7b81-4639-86f5-a8b1802a4fd9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wgx9q\" (UID: \"1fdd1cba-7b81-4639-86f5-a8b1802a4fd9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wgx9q" Sep 16 13:35:44.162599 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:44.162573 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lppt9\" (UniqueName: \"kubernetes.io/projected/1fdd1cba-7b81-4639-86f5-a8b1802a4fd9-kube-api-access-lppt9\") pod \"cert-manager-operator-controller-manager-55c444bf86-wgx9q\" (UID: \"1fdd1cba-7b81-4639-86f5-a8b1802a4fd9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wgx9q" Sep 16 13:35:44.163280 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:44.163263 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wgx9q" Sep 16 13:35:44.500007 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:44.499984 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wgx9q"] Sep 16 13:35:44.502460 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:35:44.502434 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fdd1cba_7b81_4639_86f5_a8b1802a4fd9.slice/crio-70eca3b777d1caa6af2c5376024002a8e10a842df3c3039b162bdc558f014df9 WatchSource:0}: Error finding container 70eca3b777d1caa6af2c5376024002a8e10a842df3c3039b162bdc558f014df9: Status 404 returned error can't find the container with id 70eca3b777d1caa6af2c5376024002a8e10a842df3c3039b162bdc558f014df9 Sep 16 13:35:44.505027 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:44.505011 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 13:35:44.763684 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:44.762428 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wgx9q" event={"ID":"1fdd1cba-7b81-4639-86f5-a8b1802a4fd9","Type":"ContainerStarted","Data":"70eca3b777d1caa6af2c5376024002a8e10a842df3c3039b162bdc558f014df9"} Sep 16 13:35:47.778026 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:47.777939 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wgx9q" event={"ID":"1fdd1cba-7b81-4639-86f5-a8b1802a4fd9","Type":"ContainerStarted","Data":"ce50e40e8d0ce5722069404aecbe9aff772131126c40fd99451cd0350be223a1"} Sep 16 13:35:47.812154 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:47.812094 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wgx9q" podStartSLOduration=1.809933502 podStartE2EDuration="4.812078456s" totalImagesPullingTime="3.002144954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:35:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:35:47.811226053 +0000 UTC m=+308.855529039" watchObservedRunningTime="2026-09-16 13:35:47.812078456 +0000 UTC m=+308.856381421" Sep 16 13:35:49.265758 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:49.265726 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh"] Sep 16 13:35:49.268571 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:49.268550 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh" Sep 16 13:35:49.271805 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:49.271786 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 13:35:49.271931 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:49.271828 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 13:35:49.280560 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:49.280544 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-75dbt\"" Sep 16 13:35:49.289292 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:49.289272 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c938cab-a216-4d6c-9c70-a1fa26b19967-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh\" (UID: \"7c938cab-a216-4d6c-9c70-a1fa26b19967\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh" Sep 16 13:35:49.289395 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:49.289309 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c938cab-a216-4d6c-9c70-a1fa26b19967-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh\" (UID: \"7c938cab-a216-4d6c-9c70-a1fa26b19967\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh" Sep 16 13:35:49.289395 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:49.289349 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhmt9\" (UniqueName: \"kubernetes.io/projected/7c938cab-a216-4d6c-9c70-a1fa26b19967-kube-api-access-dhmt9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh\" (UID: \"7c938cab-a216-4d6c-9c70-a1fa26b19967\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh" Sep 16 13:35:49.296128 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:49.296104 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh"] Sep 16 13:35:49.390123 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:49.390084 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhmt9\" (UniqueName: \"kubernetes.io/projected/7c938cab-a216-4d6c-9c70-a1fa26b19967-kube-api-access-dhmt9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh\" (UID: \"7c938cab-a216-4d6c-9c70-a1fa26b19967\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh" Sep 16 13:35:49.390340 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:49.390141 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c938cab-a216-4d6c-9c70-a1fa26b19967-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh\" (UID: \"7c938cab-a216-4d6c-9c70-a1fa26b19967\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh" Sep 16 13:35:49.390340 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:49.390196 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c938cab-a216-4d6c-9c70-a1fa26b19967-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh\" (UID: \"7c938cab-a216-4d6c-9c70-a1fa26b19967\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh" Sep 16 13:35:49.390581 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:49.390564 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c938cab-a216-4d6c-9c70-a1fa26b19967-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh\" (UID: \"7c938cab-a216-4d6c-9c70-a1fa26b19967\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh" Sep 16 13:35:49.390633 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:49.390581 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c938cab-a216-4d6c-9c70-a1fa26b19967-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh\" (UID: \"7c938cab-a216-4d6c-9c70-a1fa26b19967\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh" Sep 16 13:35:49.402626 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:49.402582 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhmt9\" (UniqueName: \"kubernetes.io/projected/7c938cab-a216-4d6c-9c70-a1fa26b19967-kube-api-access-dhmt9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh\" (UID: \"7c938cab-a216-4d6c-9c70-a1fa26b19967\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh" Sep 16 13:35:49.577624 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:49.577544 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh" Sep 16 13:35:49.936497 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:49.936470 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh"] Sep 16 13:35:49.937394 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:35:49.937346 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c938cab_a216_4d6c_9c70_a1fa26b19967.slice/crio-e7c9771f1b50d4dcef197f95b4f56e9e460c042d3b094260bc159f67f214acc8 WatchSource:0}: Error finding container e7c9771f1b50d4dcef197f95b4f56e9e460c042d3b094260bc159f67f214acc8: Status 404 returned error can't find the container with id e7c9771f1b50d4dcef197f95b4f56e9e460c042d3b094260bc159f67f214acc8 Sep 16 13:35:50.792080 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:50.792040 2562 generic.go:356] "Generic (PLEG): container finished" podID="7c938cab-a216-4d6c-9c70-a1fa26b19967" containerID="a0891bace1b9632317235d1d5add119ddb5d41a506ecd26055f27d4c1071b690" exitCode=0 Sep 16 13:35:50.792543 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:50.792129 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh" event={"ID":"7c938cab-a216-4d6c-9c70-a1fa26b19967","Type":"ContainerDied","Data":"a0891bace1b9632317235d1d5add119ddb5d41a506ecd26055f27d4c1071b690"} Sep 16 13:35:50.792543 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:50.792220 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh" event={"ID":"7c938cab-a216-4d6c-9c70-a1fa26b19967","Type":"ContainerStarted","Data":"e7c9771f1b50d4dcef197f95b4f56e9e460c042d3b094260bc159f67f214acc8"} Sep 16 13:35:52.803759 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:52.803723 2562 generic.go:356] "Generic (PLEG): container finished" podID="7c938cab-a216-4d6c-9c70-a1fa26b19967" containerID="d6414c5898b8b57e26e82e339838c6a38d5c8889dc3a035fd0020b57bb5d3c4f" exitCode=0 Sep 16 13:35:52.804219 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:52.803814 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh" event={"ID":"7c938cab-a216-4d6c-9c70-a1fa26b19967","Type":"ContainerDied","Data":"d6414c5898b8b57e26e82e339838c6a38d5c8889dc3a035fd0020b57bb5d3c4f"} Sep 16 13:35:53.810129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:53.810094 2562 generic.go:356] "Generic (PLEG): container finished" podID="7c938cab-a216-4d6c-9c70-a1fa26b19967" containerID="674e45c39262a1a69902df8a8a967d1a793c1566c5afca38fc63236e28f4977b" exitCode=0 Sep 16 13:35:53.810547 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:53.810166 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh" event={"ID":"7c938cab-a216-4d6c-9c70-a1fa26b19967","Type":"ContainerDied","Data":"674e45c39262a1a69902df8a8a967d1a793c1566c5afca38fc63236e28f4977b"} Sep 16 13:35:54.939366 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:54.939338 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh" Sep 16 13:35:55.033392 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.033344 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c938cab-a216-4d6c-9c70-a1fa26b19967-bundle\") pod \"7c938cab-a216-4d6c-9c70-a1fa26b19967\" (UID: \"7c938cab-a216-4d6c-9c70-a1fa26b19967\") " Sep 16 13:35:55.033553 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.033447 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhmt9\" (UniqueName: \"kubernetes.io/projected/7c938cab-a216-4d6c-9c70-a1fa26b19967-kube-api-access-dhmt9\") pod \"7c938cab-a216-4d6c-9c70-a1fa26b19967\" (UID: \"7c938cab-a216-4d6c-9c70-a1fa26b19967\") " Sep 16 13:35:55.033553 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.033526 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c938cab-a216-4d6c-9c70-a1fa26b19967-util\") pod \"7c938cab-a216-4d6c-9c70-a1fa26b19967\" (UID: \"7c938cab-a216-4d6c-9c70-a1fa26b19967\") " Sep 16 13:35:55.033802 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.033784 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c938cab-a216-4d6c-9c70-a1fa26b19967-bundle" (OuterVolumeSpecName: "bundle") pod "7c938cab-a216-4d6c-9c70-a1fa26b19967" (UID: "7c938cab-a216-4d6c-9c70-a1fa26b19967"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:35:55.035630 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.035610 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c938cab-a216-4d6c-9c70-a1fa26b19967-kube-api-access-dhmt9" (OuterVolumeSpecName: "kube-api-access-dhmt9") pod "7c938cab-a216-4d6c-9c70-a1fa26b19967" (UID: "7c938cab-a216-4d6c-9c70-a1fa26b19967"). InnerVolumeSpecName "kube-api-access-dhmt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:35:55.070135 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.070054 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c938cab-a216-4d6c-9c70-a1fa26b19967-util" (OuterVolumeSpecName: "util") pod "7c938cab-a216-4d6c-9c70-a1fa26b19967" (UID: "7c938cab-a216-4d6c-9c70-a1fa26b19967"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:35:55.122120 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.122083 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-79stx"] Sep 16 13:35:55.122448 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.122436 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c938cab-a216-4d6c-9c70-a1fa26b19967" containerName="util" Sep 16 13:35:55.122497 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.122450 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c938cab-a216-4d6c-9c70-a1fa26b19967" containerName="util" Sep 16 13:35:55.122497 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.122467 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c938cab-a216-4d6c-9c70-a1fa26b19967" containerName="pull" Sep 16 13:35:55.122497 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.122472 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c938cab-a216-4d6c-9c70-a1fa26b19967" containerName="pull" Sep 16 13:35:55.122497 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.122484 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c938cab-a216-4d6c-9c70-a1fa26b19967" containerName="extract" Sep 16 13:35:55.122497 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.122491 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c938cab-a216-4d6c-9c70-a1fa26b19967" containerName="extract" Sep 16 13:35:55.122637 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.122545 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c938cab-a216-4d6c-9c70-a1fa26b19967" containerName="extract" Sep 16 13:35:55.124421 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.124404 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-79stx" Sep 16 13:35:55.129229 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.129206 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 16 13:35:55.129395 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.129362 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-8clwx\"" Sep 16 13:35:55.129465 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.129451 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 16 13:35:55.134173 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.134158 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c938cab-a216-4d6c-9c70-a1fa26b19967-util\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:35:55.134173 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.134174 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c938cab-a216-4d6c-9c70-a1fa26b19967-bundle\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:35:55.134271 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.134183 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhmt9\" (UniqueName: \"kubernetes.io/projected/7c938cab-a216-4d6c-9c70-a1fa26b19967-kube-api-access-dhmt9\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:35:55.139436 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.139405 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-79stx"] Sep 16 13:35:55.234532 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.234492 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8b45\" (UniqueName: \"kubernetes.io/projected/e29a1118-b042-49f3-85db-ec0bb5cde29a-kube-api-access-k8b45\") pod \"cert-manager-cainjector-559cdd8c87-79stx\" (UID: \"e29a1118-b042-49f3-85db-ec0bb5cde29a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-79stx" Sep 16 13:35:55.234702 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.234559 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e29a1118-b042-49f3-85db-ec0bb5cde29a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-79stx\" (UID: \"e29a1118-b042-49f3-85db-ec0bb5cde29a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-79stx" Sep 16 13:35:55.335221 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.335131 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e29a1118-b042-49f3-85db-ec0bb5cde29a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-79stx\" (UID: \"e29a1118-b042-49f3-85db-ec0bb5cde29a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-79stx" Sep 16 13:35:55.335221 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.335186 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8b45\" (UniqueName: \"kubernetes.io/projected/e29a1118-b042-49f3-85db-ec0bb5cde29a-kube-api-access-k8b45\") pod \"cert-manager-cainjector-559cdd8c87-79stx\" (UID: \"e29a1118-b042-49f3-85db-ec0bb5cde29a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-79stx" Sep 16 13:35:55.344744 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.344710 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e29a1118-b042-49f3-85db-ec0bb5cde29a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-79stx\" (UID: \"e29a1118-b042-49f3-85db-ec0bb5cde29a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-79stx" Sep 16 13:35:55.344868 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.344760 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8b45\" (UniqueName: \"kubernetes.io/projected/e29a1118-b042-49f3-85db-ec0bb5cde29a-kube-api-access-k8b45\") pod \"cert-manager-cainjector-559cdd8c87-79stx\" (UID: \"e29a1118-b042-49f3-85db-ec0bb5cde29a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-79stx" Sep 16 13:35:55.454266 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.454229 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-79stx" Sep 16 13:35:55.725726 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.725705 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-79stx"] Sep 16 13:35:55.728128 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:35:55.728099 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode29a1118_b042_49f3_85db_ec0bb5cde29a.slice/crio-426aeb9dd64e86d4a7c21b0285028550a4e6871796a1ff261833573cba21a849 WatchSource:0}: Error finding container 426aeb9dd64e86d4a7c21b0285028550a4e6871796a1ff261833573cba21a849: Status 404 returned error can't find the container with id 426aeb9dd64e86d4a7c21b0285028550a4e6871796a1ff261833573cba21a849 Sep 16 13:35:55.825573 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.825542 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh" Sep 16 13:35:55.825573 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.825554 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvt7kh" event={"ID":"7c938cab-a216-4d6c-9c70-a1fa26b19967","Type":"ContainerDied","Data":"e7c9771f1b50d4dcef197f95b4f56e9e460c042d3b094260bc159f67f214acc8"} Sep 16 13:35:55.825781 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.825633 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7c9771f1b50d4dcef197f95b4f56e9e460c042d3b094260bc159f67f214acc8" Sep 16 13:35:55.827074 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:55.827049 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-79stx" event={"ID":"e29a1118-b042-49f3-85db-ec0bb5cde29a","Type":"ContainerStarted","Data":"426aeb9dd64e86d4a7c21b0285028550a4e6871796a1ff261833573cba21a849"} Sep 16 13:35:59.150674 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:59.150592 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-p59q6"] Sep 16 13:35:59.153292 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:59.153272 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-p59q6" Sep 16 13:35:59.157150 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:59.157129 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-zjh8l\"" Sep 16 13:35:59.166454 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:59.166433 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-p59q6"] Sep 16 13:35:59.169250 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:59.169227 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/71f5093e-0778-4bd4-a988-a006f27c94ce-bound-sa-token\") pod \"cert-manager-55b4d954d5-p59q6\" (UID: \"71f5093e-0778-4bd4-a988-a006f27c94ce\") " pod="cert-manager/cert-manager-55b4d954d5-p59q6" Sep 16 13:35:59.169371 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:59.169351 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zchz\" (UniqueName: \"kubernetes.io/projected/71f5093e-0778-4bd4-a988-a006f27c94ce-kube-api-access-8zchz\") pod \"cert-manager-55b4d954d5-p59q6\" (UID: \"71f5093e-0778-4bd4-a988-a006f27c94ce\") " pod="cert-manager/cert-manager-55b4d954d5-p59q6" Sep 16 13:35:59.270260 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:59.270225 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zchz\" (UniqueName: \"kubernetes.io/projected/71f5093e-0778-4bd4-a988-a006f27c94ce-kube-api-access-8zchz\") pod \"cert-manager-55b4d954d5-p59q6\" (UID: \"71f5093e-0778-4bd4-a988-a006f27c94ce\") " pod="cert-manager/cert-manager-55b4d954d5-p59q6" Sep 16 13:35:59.270260 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:59.270277 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/71f5093e-0778-4bd4-a988-a006f27c94ce-bound-sa-token\") pod \"cert-manager-55b4d954d5-p59q6\" (UID: \"71f5093e-0778-4bd4-a988-a006f27c94ce\") " pod="cert-manager/cert-manager-55b4d954d5-p59q6" Sep 16 13:35:59.282552 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:59.282527 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zchz\" (UniqueName: \"kubernetes.io/projected/71f5093e-0778-4bd4-a988-a006f27c94ce-kube-api-access-8zchz\") pod \"cert-manager-55b4d954d5-p59q6\" (UID: \"71f5093e-0778-4bd4-a988-a006f27c94ce\") " pod="cert-manager/cert-manager-55b4d954d5-p59q6" Sep 16 13:35:59.282669 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:59.282564 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/71f5093e-0778-4bd4-a988-a006f27c94ce-bound-sa-token\") pod \"cert-manager-55b4d954d5-p59q6\" (UID: \"71f5093e-0778-4bd4-a988-a006f27c94ce\") " pod="cert-manager/cert-manager-55b4d954d5-p59q6" Sep 16 13:35:59.463320 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:59.463287 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-p59q6" Sep 16 13:35:59.736081 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:59.736059 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-p59q6"] Sep 16 13:35:59.738073 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:35:59.738048 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71f5093e_0778_4bd4_a988_a006f27c94ce.slice/crio-371cc65a16d7c5e9e9cfbb4b0be1338bdf0f4510dfbd5bc255e89de3d1e8f0b2 WatchSource:0}: Error finding container 371cc65a16d7c5e9e9cfbb4b0be1338bdf0f4510dfbd5bc255e89de3d1e8f0b2: Status 404 returned error can't find the container with id 371cc65a16d7c5e9e9cfbb4b0be1338bdf0f4510dfbd5bc255e89de3d1e8f0b2 Sep 16 13:35:59.847146 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:59.847111 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-79stx" event={"ID":"e29a1118-b042-49f3-85db-ec0bb5cde29a","Type":"ContainerStarted","Data":"2da0cc739deb06f91631664fcb3986302e56519db77abe034bf8efbd21a8cf78"} Sep 16 13:35:59.848855 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:59.848827 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-p59q6" event={"ID":"71f5093e-0778-4bd4-a988-a006f27c94ce","Type":"ContainerStarted","Data":"e9fc3f8be00c091df7f2a2877d9b330f2a281b1da39883b974ff265d63fa8786"} Sep 16 13:35:59.848962 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:59.848866 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-p59q6" event={"ID":"71f5093e-0778-4bd4-a988-a006f27c94ce","Type":"ContainerStarted","Data":"371cc65a16d7c5e9e9cfbb4b0be1338bdf0f4510dfbd5bc255e89de3d1e8f0b2"} Sep 16 13:35:59.871925 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:59.871876 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-79stx" podStartSLOduration=1.794919299 podStartE2EDuration="4.871860351s" totalImagesPullingTime="3.076941052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:35:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:35:59.870699734 +0000 UTC m=+320.915002701" watchObservedRunningTime="2026-09-16 13:35:59.871860351 +0000 UTC m=+320.916163317" Sep 16 13:35:59.893232 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:35:59.893184 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-p59q6" podStartSLOduration=0.893166997 podStartE2EDuration="893.166997ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:35:59.890760444 +0000 UTC m=+320.935063410" watchObservedRunningTime="2026-09-16 13:35:59.893166997 +0000 UTC m=+320.937469961" Sep 16 13:36:12.747334 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:12.747299 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp"] Sep 16 13:36:12.751436 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:12.751414 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp" Sep 16 13:36:12.754419 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:12.754357 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 13:36:12.754560 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:12.754542 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-75dbt\"" Sep 16 13:36:12.755665 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:12.755644 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 13:36:12.762819 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:12.762799 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp"] Sep 16 13:36:12.881722 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:12.881683 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da969c5a-d4f1-4c87-96ee-4be47c43c671-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp\" (UID: \"da969c5a-d4f1-4c87-96ee-4be47c43c671\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp" Sep 16 13:36:12.881897 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:12.881757 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klnfx\" (UniqueName: \"kubernetes.io/projected/da969c5a-d4f1-4c87-96ee-4be47c43c671-kube-api-access-klnfx\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp\" (UID: \"da969c5a-d4f1-4c87-96ee-4be47c43c671\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp" Sep 16 13:36:12.881897 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:12.881803 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da969c5a-d4f1-4c87-96ee-4be47c43c671-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp\" (UID: \"da969c5a-d4f1-4c87-96ee-4be47c43c671\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp" Sep 16 13:36:12.982472 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:12.982427 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da969c5a-d4f1-4c87-96ee-4be47c43c671-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp\" (UID: \"da969c5a-d4f1-4c87-96ee-4be47c43c671\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp" Sep 16 13:36:12.982643 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:12.982479 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da969c5a-d4f1-4c87-96ee-4be47c43c671-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp\" (UID: \"da969c5a-d4f1-4c87-96ee-4be47c43c671\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp" Sep 16 13:36:12.982643 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:12.982532 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klnfx\" (UniqueName: \"kubernetes.io/projected/da969c5a-d4f1-4c87-96ee-4be47c43c671-kube-api-access-klnfx\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp\" (UID: \"da969c5a-d4f1-4c87-96ee-4be47c43c671\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp" Sep 16 13:36:12.982928 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:12.982905 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da969c5a-d4f1-4c87-96ee-4be47c43c671-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp\" (UID: \"da969c5a-d4f1-4c87-96ee-4be47c43c671\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp" Sep 16 13:36:12.982975 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:12.982903 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da969c5a-d4f1-4c87-96ee-4be47c43c671-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp\" (UID: \"da969c5a-d4f1-4c87-96ee-4be47c43c671\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp" Sep 16 13:36:12.991415 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:12.991361 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klnfx\" (UniqueName: \"kubernetes.io/projected/da969c5a-d4f1-4c87-96ee-4be47c43c671-kube-api-access-klnfx\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp\" (UID: \"da969c5a-d4f1-4c87-96ee-4be47c43c671\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp" Sep 16 13:36:13.061462 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:13.061364 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp" Sep 16 13:36:13.334609 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:13.334581 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp"] Sep 16 13:36:13.337202 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:36:13.337174 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda969c5a_d4f1_4c87_96ee_4be47c43c671.slice/crio-761043958dcd1262551b0f9c4a0657872cdab9d99554563d9d7098493cd9a6d5 WatchSource:0}: Error finding container 761043958dcd1262551b0f9c4a0657872cdab9d99554563d9d7098493cd9a6d5: Status 404 returned error can't find the container with id 761043958dcd1262551b0f9c4a0657872cdab9d99554563d9d7098493cd9a6d5 Sep 16 13:36:13.917104 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:13.917068 2562 generic.go:356] "Generic (PLEG): container finished" podID="da969c5a-d4f1-4c87-96ee-4be47c43c671" containerID="d7e191b5b76a1c43b9cf8307455b3731c9b2596ea0d1674115a8fb087c970757" exitCode=0 Sep 16 13:36:13.917610 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:13.917158 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp" event={"ID":"da969c5a-d4f1-4c87-96ee-4be47c43c671","Type":"ContainerDied","Data":"d7e191b5b76a1c43b9cf8307455b3731c9b2596ea0d1674115a8fb087c970757"} Sep 16 13:36:13.917610 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:13.917208 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp" event={"ID":"da969c5a-d4f1-4c87-96ee-4be47c43c671","Type":"ContainerStarted","Data":"761043958dcd1262551b0f9c4a0657872cdab9d99554563d9d7098493cd9a6d5"} Sep 16 13:36:14.924137 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:14.924108 2562 generic.go:356] "Generic (PLEG): container finished" podID="da969c5a-d4f1-4c87-96ee-4be47c43c671" containerID="59b5c18df137c6fc635ea030b13179e51ae7329981e3c03e2461dff9e42c1266" exitCode=0 Sep 16 13:36:14.924552 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:14.924163 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp" event={"ID":"da969c5a-d4f1-4c87-96ee-4be47c43c671","Type":"ContainerDied","Data":"59b5c18df137c6fc635ea030b13179e51ae7329981e3c03e2461dff9e42c1266"} Sep 16 13:36:15.930960 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:15.930922 2562 generic.go:356] "Generic (PLEG): container finished" podID="da969c5a-d4f1-4c87-96ee-4be47c43c671" containerID="5b1bc64995e45d7796504901bde3236982478d597739f248e07ac89afc8a6ee2" exitCode=0 Sep 16 13:36:15.930960 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:15.930960 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp" event={"ID":"da969c5a-d4f1-4c87-96ee-4be47c43c671","Type":"ContainerDied","Data":"5b1bc64995e45d7796504901bde3236982478d597739f248e07ac89afc8a6ee2"} Sep 16 13:36:17.065979 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:17.065952 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp" Sep 16 13:36:17.118428 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:17.118367 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da969c5a-d4f1-4c87-96ee-4be47c43c671-bundle\") pod \"da969c5a-d4f1-4c87-96ee-4be47c43c671\" (UID: \"da969c5a-d4f1-4c87-96ee-4be47c43c671\") " Sep 16 13:36:17.118578 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:17.118445 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da969c5a-d4f1-4c87-96ee-4be47c43c671-util\") pod \"da969c5a-d4f1-4c87-96ee-4be47c43c671\" (UID: \"da969c5a-d4f1-4c87-96ee-4be47c43c671\") " Sep 16 13:36:17.118578 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:17.118474 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klnfx\" (UniqueName: \"kubernetes.io/projected/da969c5a-d4f1-4c87-96ee-4be47c43c671-kube-api-access-klnfx\") pod \"da969c5a-d4f1-4c87-96ee-4be47c43c671\" (UID: \"da969c5a-d4f1-4c87-96ee-4be47c43c671\") " Sep 16 13:36:17.119026 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:17.119003 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da969c5a-d4f1-4c87-96ee-4be47c43c671-bundle" (OuterVolumeSpecName: "bundle") pod "da969c5a-d4f1-4c87-96ee-4be47c43c671" (UID: "da969c5a-d4f1-4c87-96ee-4be47c43c671"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:36:17.120822 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:17.120799 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da969c5a-d4f1-4c87-96ee-4be47c43c671-kube-api-access-klnfx" (OuterVolumeSpecName: "kube-api-access-klnfx") pod "da969c5a-d4f1-4c87-96ee-4be47c43c671" (UID: "da969c5a-d4f1-4c87-96ee-4be47c43c671"). InnerVolumeSpecName "kube-api-access-klnfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:36:17.123666 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:17.123624 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da969c5a-d4f1-4c87-96ee-4be47c43c671-util" (OuterVolumeSpecName: "util") pod "da969c5a-d4f1-4c87-96ee-4be47c43c671" (UID: "da969c5a-d4f1-4c87-96ee-4be47c43c671"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:36:17.219811 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:17.219779 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da969c5a-d4f1-4c87-96ee-4be47c43c671-bundle\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:36:17.219811 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:17.219808 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da969c5a-d4f1-4c87-96ee-4be47c43c671-util\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:36:17.219811 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:17.219817 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klnfx\" (UniqueName: \"kubernetes.io/projected/da969c5a-d4f1-4c87-96ee-4be47c43c671-kube-api-access-klnfx\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:36:17.941786 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:17.941749 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp" event={"ID":"da969c5a-d4f1-4c87-96ee-4be47c43c671","Type":"ContainerDied","Data":"761043958dcd1262551b0f9c4a0657872cdab9d99554563d9d7098493cd9a6d5"} Sep 16 13:36:17.941786 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:17.941770 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvlhtp" Sep 16 13:36:17.941786 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:17.941795 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="761043958dcd1262551b0f9c4a0657872cdab9d99554563d9d7098493cd9a6d5" Sep 16 13:36:22.801081 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:22.801041 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb"] Sep 16 13:36:22.801585 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:22.801354 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da969c5a-d4f1-4c87-96ee-4be47c43c671" containerName="util" Sep 16 13:36:22.801585 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:22.801365 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="da969c5a-d4f1-4c87-96ee-4be47c43c671" containerName="util" Sep 16 13:36:22.801585 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:22.801388 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da969c5a-d4f1-4c87-96ee-4be47c43c671" containerName="pull" Sep 16 13:36:22.801585 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:22.801394 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="da969c5a-d4f1-4c87-96ee-4be47c43c671" containerName="pull" Sep 16 13:36:22.801585 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:22.801401 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da969c5a-d4f1-4c87-96ee-4be47c43c671" containerName="extract" Sep 16 13:36:22.801585 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:22.801406 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="da969c5a-d4f1-4c87-96ee-4be47c43c671" containerName="extract" Sep 16 13:36:22.801585 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:22.801467 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="da969c5a-d4f1-4c87-96ee-4be47c43c671" containerName="extract" Sep 16 13:36:22.807879 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:22.807860 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb" Sep 16 13:36:22.813127 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:22.813107 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-75dbt\"" Sep 16 13:36:22.813575 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:22.813551 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 13:36:22.814304 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:22.814288 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 13:36:22.819625 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:22.819598 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb"] Sep 16 13:36:22.869889 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:22.869857 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6957309c-b213-4aaf-ac7e-6725f64fa446-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb\" (UID: \"6957309c-b213-4aaf-ac7e-6725f64fa446\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb" Sep 16 13:36:22.870059 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:22.869900 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6957309c-b213-4aaf-ac7e-6725f64fa446-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb\" (UID: \"6957309c-b213-4aaf-ac7e-6725f64fa446\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb" Sep 16 13:36:22.870059 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:22.869942 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pq84\" (UniqueName: \"kubernetes.io/projected/6957309c-b213-4aaf-ac7e-6725f64fa446-kube-api-access-8pq84\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb\" (UID: \"6957309c-b213-4aaf-ac7e-6725f64fa446\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb" Sep 16 13:36:22.970797 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:22.970766 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6957309c-b213-4aaf-ac7e-6725f64fa446-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb\" (UID: \"6957309c-b213-4aaf-ac7e-6725f64fa446\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb" Sep 16 13:36:22.970971 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:22.970813 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6957309c-b213-4aaf-ac7e-6725f64fa446-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb\" (UID: \"6957309c-b213-4aaf-ac7e-6725f64fa446\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb" Sep 16 13:36:22.970971 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:22.970855 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pq84\" (UniqueName: \"kubernetes.io/projected/6957309c-b213-4aaf-ac7e-6725f64fa446-kube-api-access-8pq84\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb\" (UID: \"6957309c-b213-4aaf-ac7e-6725f64fa446\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb" Sep 16 13:36:22.971137 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:22.971119 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6957309c-b213-4aaf-ac7e-6725f64fa446-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb\" (UID: \"6957309c-b213-4aaf-ac7e-6725f64fa446\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb" Sep 16 13:36:22.971187 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:22.971171 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6957309c-b213-4aaf-ac7e-6725f64fa446-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb\" (UID: \"6957309c-b213-4aaf-ac7e-6725f64fa446\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb" Sep 16 13:36:22.982565 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:22.982544 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pq84\" (UniqueName: \"kubernetes.io/projected/6957309c-b213-4aaf-ac7e-6725f64fa446-kube-api-access-8pq84\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb\" (UID: \"6957309c-b213-4aaf-ac7e-6725f64fa446\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb" Sep 16 13:36:23.116156 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:23.116066 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb" Sep 16 13:36:23.403878 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:23.403852 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb"] Sep 16 13:36:23.405356 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:36:23.405331 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6957309c_b213_4aaf_ac7e_6725f64fa446.slice/crio-438c14078a5f8623483284fc3c58043467a1f6dfdf1920e613bf5e978c39cd8f WatchSource:0}: Error finding container 438c14078a5f8623483284fc3c58043467a1f6dfdf1920e613bf5e978c39cd8f: Status 404 returned error can't find the container with id 438c14078a5f8623483284fc3c58043467a1f6dfdf1920e613bf5e978c39cd8f Sep 16 13:36:23.971041 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:23.971004 2562 generic.go:356] "Generic (PLEG): container finished" podID="6957309c-b213-4aaf-ac7e-6725f64fa446" containerID="94ab544e6263bb3adf926de72973561881162ab40c15f4297e93bdd85c337d2d" exitCode=0 Sep 16 13:36:23.971443 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:23.971121 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb" event={"ID":"6957309c-b213-4aaf-ac7e-6725f64fa446","Type":"ContainerDied","Data":"94ab544e6263bb3adf926de72973561881162ab40c15f4297e93bdd85c337d2d"} Sep 16 13:36:23.971443 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:23.971171 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb" event={"ID":"6957309c-b213-4aaf-ac7e-6725f64fa446","Type":"ContainerStarted","Data":"438c14078a5f8623483284fc3c58043467a1f6dfdf1920e613bf5e978c39cd8f"} Sep 16 13:36:24.978047 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:24.978016 2562 generic.go:356] "Generic (PLEG): container finished" podID="6957309c-b213-4aaf-ac7e-6725f64fa446" containerID="7f1a081db15b905e375ab195c61436bd714238186738fcbfb507e05b1dd8c300" exitCode=0 Sep 16 13:36:24.978476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:24.978052 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb" event={"ID":"6957309c-b213-4aaf-ac7e-6725f64fa446","Type":"ContainerDied","Data":"7f1a081db15b905e375ab195c61436bd714238186738fcbfb507e05b1dd8c300"} Sep 16 13:36:25.984640 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:25.984600 2562 generic.go:356] "Generic (PLEG): container finished" podID="6957309c-b213-4aaf-ac7e-6725f64fa446" containerID="32972d9907318243b153b2f2ac89c3fe15037aff66ea0082548767d58d42250e" exitCode=0 Sep 16 13:36:25.985005 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:25.984681 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb" event={"ID":"6957309c-b213-4aaf-ac7e-6725f64fa446","Type":"ContainerDied","Data":"32972d9907318243b153b2f2ac89c3fe15037aff66ea0082548767d58d42250e"} Sep 16 13:36:26.987198 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:26.987162 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5"] Sep 16 13:36:26.990570 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:26.990552 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5" Sep 16 13:36:26.995955 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:26.995930 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 16 13:36:26.995955 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:26.995946 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 16 13:36:26.996914 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:26.996887 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 16 13:36:26.996914 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:26.996890 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 16 13:36:27.001717 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.001697 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2tk7k\"" Sep 16 13:36:27.011515 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.011486 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5"] Sep 16 13:36:27.108974 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.108941 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24c731c4-8188-4a8b-ae14-34aa43244e9b-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5\" (UID: \"24c731c4-8188-4a8b-ae14-34aa43244e9b\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5" Sep 16 13:36:27.109089 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.108975 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/24c731c4-8188-4a8b-ae14-34aa43244e9b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5\" (UID: \"24c731c4-8188-4a8b-ae14-34aa43244e9b\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5" Sep 16 13:36:27.109089 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.109013 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24c731c4-8188-4a8b-ae14-34aa43244e9b-tmp\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5\" (UID: \"24c731c4-8188-4a8b-ae14-34aa43244e9b\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5" Sep 16 13:36:27.109089 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.109051 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/24c731c4-8188-4a8b-ae14-34aa43244e9b-manifests\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5\" (UID: \"24c731c4-8188-4a8b-ae14-34aa43244e9b\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5" Sep 16 13:36:27.109182 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.109108 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksrc8\" (UniqueName: \"kubernetes.io/projected/24c731c4-8188-4a8b-ae14-34aa43244e9b-kube-api-access-ksrc8\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5\" (UID: \"24c731c4-8188-4a8b-ae14-34aa43244e9b\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5" Sep 16 13:36:27.115051 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.115031 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb" Sep 16 13:36:27.209855 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.209822 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pq84\" (UniqueName: \"kubernetes.io/projected/6957309c-b213-4aaf-ac7e-6725f64fa446-kube-api-access-8pq84\") pod \"6957309c-b213-4aaf-ac7e-6725f64fa446\" (UID: \"6957309c-b213-4aaf-ac7e-6725f64fa446\") " Sep 16 13:36:27.210019 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.209875 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6957309c-b213-4aaf-ac7e-6725f64fa446-util\") pod \"6957309c-b213-4aaf-ac7e-6725f64fa446\" (UID: \"6957309c-b213-4aaf-ac7e-6725f64fa446\") " Sep 16 13:36:27.210019 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.209895 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6957309c-b213-4aaf-ac7e-6725f64fa446-bundle\") pod \"6957309c-b213-4aaf-ac7e-6725f64fa446\" (UID: \"6957309c-b213-4aaf-ac7e-6725f64fa446\") " Sep 16 13:36:27.210100 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.210019 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24c731c4-8188-4a8b-ae14-34aa43244e9b-tmp\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5\" (UID: \"24c731c4-8188-4a8b-ae14-34aa43244e9b\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5" Sep 16 13:36:27.210100 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.210041 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/24c731c4-8188-4a8b-ae14-34aa43244e9b-manifests\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5\" (UID: \"24c731c4-8188-4a8b-ae14-34aa43244e9b\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5" Sep 16 13:36:27.210100 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.210084 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksrc8\" (UniqueName: \"kubernetes.io/projected/24c731c4-8188-4a8b-ae14-34aa43244e9b-kube-api-access-ksrc8\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5\" (UID: \"24c731c4-8188-4a8b-ae14-34aa43244e9b\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5" Sep 16 13:36:27.210250 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.210136 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24c731c4-8188-4a8b-ae14-34aa43244e9b-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5\" (UID: \"24c731c4-8188-4a8b-ae14-34aa43244e9b\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5" Sep 16 13:36:27.210250 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.210183 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/24c731c4-8188-4a8b-ae14-34aa43244e9b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5\" (UID: \"24c731c4-8188-4a8b-ae14-34aa43244e9b\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5" Sep 16 13:36:27.210609 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.210534 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/24c731c4-8188-4a8b-ae14-34aa43244e9b-manifests\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5\" (UID: \"24c731c4-8188-4a8b-ae14-34aa43244e9b\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5" Sep 16 13:36:27.211066 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.211036 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6957309c-b213-4aaf-ac7e-6725f64fa446-bundle" (OuterVolumeSpecName: "bundle") pod "6957309c-b213-4aaf-ac7e-6725f64fa446" (UID: "6957309c-b213-4aaf-ac7e-6725f64fa446"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:36:27.212217 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.212194 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6957309c-b213-4aaf-ac7e-6725f64fa446-kube-api-access-8pq84" (OuterVolumeSpecName: "kube-api-access-8pq84") pod "6957309c-b213-4aaf-ac7e-6725f64fa446" (UID: "6957309c-b213-4aaf-ac7e-6725f64fa446"). InnerVolumeSpecName "kube-api-access-8pq84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:36:27.212495 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.212416 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24c731c4-8188-4a8b-ae14-34aa43244e9b-tmp\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5\" (UID: \"24c731c4-8188-4a8b-ae14-34aa43244e9b\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5" Sep 16 13:36:27.213208 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.213186 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/24c731c4-8188-4a8b-ae14-34aa43244e9b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5\" (UID: \"24c731c4-8188-4a8b-ae14-34aa43244e9b\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5" Sep 16 13:36:27.213308 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.213275 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24c731c4-8188-4a8b-ae14-34aa43244e9b-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5\" (UID: \"24c731c4-8188-4a8b-ae14-34aa43244e9b\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5" Sep 16 13:36:27.216070 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.216037 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6957309c-b213-4aaf-ac7e-6725f64fa446-util" (OuterVolumeSpecName: "util") pod "6957309c-b213-4aaf-ac7e-6725f64fa446" (UID: "6957309c-b213-4aaf-ac7e-6725f64fa446"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:36:27.227871 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.227846 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksrc8\" (UniqueName: \"kubernetes.io/projected/24c731c4-8188-4a8b-ae14-34aa43244e9b-kube-api-access-ksrc8\") pod \"opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5\" (UID: \"24c731c4-8188-4a8b-ae14-34aa43244e9b\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5" Sep 16 13:36:27.301938 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.301846 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5" Sep 16 13:36:27.311079 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.311059 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pq84\" (UniqueName: \"kubernetes.io/projected/6957309c-b213-4aaf-ac7e-6725f64fa446-kube-api-access-8pq84\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:36:27.311171 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.311080 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6957309c-b213-4aaf-ac7e-6725f64fa446-util\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:36:27.311171 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.311091 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6957309c-b213-4aaf-ac7e-6725f64fa446-bundle\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:36:27.590062 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.590041 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5"] Sep 16 13:36:27.592184 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:36:27.592149 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c731c4_8188_4a8b_ae14_34aa43244e9b.slice/crio-4d68f01419d1508c55e2c27113defd5de7d3b28be7407f6aee16d8074923fe24 WatchSource:0}: Error finding container 4d68f01419d1508c55e2c27113defd5de7d3b28be7407f6aee16d8074923fe24: Status 404 returned error can't find the container with id 4d68f01419d1508c55e2c27113defd5de7d3b28be7407f6aee16d8074923fe24 Sep 16 13:36:27.995190 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.995153 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5" event={"ID":"24c731c4-8188-4a8b-ae14-34aa43244e9b","Type":"ContainerStarted","Data":"4d68f01419d1508c55e2c27113defd5de7d3b28be7407f6aee16d8074923fe24"} Sep 16 13:36:27.997088 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.997063 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb" event={"ID":"6957309c-b213-4aaf-ac7e-6725f64fa446","Type":"ContainerDied","Data":"438c14078a5f8623483284fc3c58043467a1f6dfdf1920e613bf5e978c39cd8f"} Sep 16 13:36:27.997211 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.997094 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jwgzb" Sep 16 13:36:27.997211 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:27.997107 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="438c14078a5f8623483284fc3c58043467a1f6dfdf1920e613bf5e978c39cd8f" Sep 16 13:36:31.014232 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:31.014191 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5" event={"ID":"24c731c4-8188-4a8b-ae14-34aa43244e9b","Type":"ContainerStarted","Data":"cb8a1ac703572acd130d2c34db533d6dc1c1c130e9e976a32a7f303d53dffee9"} Sep 16 13:36:32.021242 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:32.021207 2562 generic.go:356] "Generic (PLEG): container finished" podID="24c731c4-8188-4a8b-ae14-34aa43244e9b" containerID="cb8a1ac703572acd130d2c34db533d6dc1c1c130e9e976a32a7f303d53dffee9" exitCode=0 Sep 16 13:36:32.021632 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:32.021263 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5" event={"ID":"24c731c4-8188-4a8b-ae14-34aa43244e9b","Type":"ContainerDied","Data":"cb8a1ac703572acd130d2c34db533d6dc1c1c130e9e976a32a7f303d53dffee9"} Sep 16 13:36:33.028252 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:33.028210 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5" event={"ID":"24c731c4-8188-4a8b-ae14-34aa43244e9b","Type":"ContainerStarted","Data":"b24b2ad6b021f5ca0ba43eb29a1dad42486d5f65cad855a31f2fa03cf39b06d1"} Sep 16 13:36:33.028783 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:33.028289 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5" Sep 16 13:36:33.069544 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:33.069494 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5" podStartSLOduration=2.631705932 podStartE2EDuration="7.069477329s" totalImagesPullingTime="3.407378775s" totalInitContainerRuntime="1.030392622s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:36:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:36:33.067745351 +0000 UTC m=+354.112048337" watchObservedRunningTime="2026-09-16 13:36:33.069477329 +0000 UTC m=+354.113780293" Sep 16 13:36:44.035127 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:44.035094 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5" Sep 16 13:36:45.088758 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.088732 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8dd669f78-gbkxs"] Sep 16 13:36:45.089129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.089041 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6957309c-b213-4aaf-ac7e-6725f64fa446" containerName="util" Sep 16 13:36:45.089129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.089051 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6957309c-b213-4aaf-ac7e-6725f64fa446" containerName="util" Sep 16 13:36:45.089129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.089071 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6957309c-b213-4aaf-ac7e-6725f64fa446" containerName="pull" Sep 16 13:36:45.089129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.089076 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6957309c-b213-4aaf-ac7e-6725f64fa446" containerName="pull" Sep 16 13:36:45.089129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.089086 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6957309c-b213-4aaf-ac7e-6725f64fa446" containerName="extract" Sep 16 13:36:45.089129 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.089091 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6957309c-b213-4aaf-ac7e-6725f64fa446" containerName="extract" Sep 16 13:36:45.089308 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.089144 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6957309c-b213-4aaf-ac7e-6725f64fa446" containerName="extract" Sep 16 13:36:45.094801 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.094781 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8dd669f78-gbkxs" Sep 16 13:36:45.098564 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.098544 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 16 13:36:45.098760 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.098748 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 16 13:36:45.099867 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.099845 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 16 13:36:45.100204 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.100187 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 16 13:36:45.100310 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.100225 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 16 13:36:45.100310 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.100259 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-x64lh\"" Sep 16 13:36:45.118400 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.118364 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8dd669f78-gbkxs"] Sep 16 13:36:45.172236 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.172204 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a21e8431-1dd2-4e8b-9acb-fb420faae4f4-manager-config\") pod \"lws-controller-manager-8dd669f78-gbkxs\" (UID: \"a21e8431-1dd2-4e8b-9acb-fb420faae4f4\") " pod="openshift-lws-operator/lws-controller-manager-8dd669f78-gbkxs" Sep 16 13:36:45.172400 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.172245 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a21e8431-1dd2-4e8b-9acb-fb420faae4f4-metrics-cert\") pod \"lws-controller-manager-8dd669f78-gbkxs\" (UID: \"a21e8431-1dd2-4e8b-9acb-fb420faae4f4\") " pod="openshift-lws-operator/lws-controller-manager-8dd669f78-gbkxs" Sep 16 13:36:45.172400 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.172312 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a21e8431-1dd2-4e8b-9acb-fb420faae4f4-cert\") pod \"lws-controller-manager-8dd669f78-gbkxs\" (UID: \"a21e8431-1dd2-4e8b-9acb-fb420faae4f4\") " pod="openshift-lws-operator/lws-controller-manager-8dd669f78-gbkxs" Sep 16 13:36:45.172400 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.172366 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf6hb\" (UniqueName: \"kubernetes.io/projected/a21e8431-1dd2-4e8b-9acb-fb420faae4f4-kube-api-access-vf6hb\") pod \"lws-controller-manager-8dd669f78-gbkxs\" (UID: \"a21e8431-1dd2-4e8b-9acb-fb420faae4f4\") " pod="openshift-lws-operator/lws-controller-manager-8dd669f78-gbkxs" Sep 16 13:36:45.273845 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.273801 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a21e8431-1dd2-4e8b-9acb-fb420faae4f4-cert\") pod \"lws-controller-manager-8dd669f78-gbkxs\" (UID: \"a21e8431-1dd2-4e8b-9acb-fb420faae4f4\") " pod="openshift-lws-operator/lws-controller-manager-8dd669f78-gbkxs" Sep 16 13:36:45.274015 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.273892 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf6hb\" (UniqueName: \"kubernetes.io/projected/a21e8431-1dd2-4e8b-9acb-fb420faae4f4-kube-api-access-vf6hb\") pod \"lws-controller-manager-8dd669f78-gbkxs\" (UID: \"a21e8431-1dd2-4e8b-9acb-fb420faae4f4\") " pod="openshift-lws-operator/lws-controller-manager-8dd669f78-gbkxs" Sep 16 13:36:45.274015 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.273930 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a21e8431-1dd2-4e8b-9acb-fb420faae4f4-manager-config\") pod \"lws-controller-manager-8dd669f78-gbkxs\" (UID: \"a21e8431-1dd2-4e8b-9acb-fb420faae4f4\") " pod="openshift-lws-operator/lws-controller-manager-8dd669f78-gbkxs" Sep 16 13:36:45.274015 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.273974 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a21e8431-1dd2-4e8b-9acb-fb420faae4f4-metrics-cert\") pod \"lws-controller-manager-8dd669f78-gbkxs\" (UID: \"a21e8431-1dd2-4e8b-9acb-fb420faae4f4\") " pod="openshift-lws-operator/lws-controller-manager-8dd669f78-gbkxs" Sep 16 13:36:45.274728 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.274680 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a21e8431-1dd2-4e8b-9acb-fb420faae4f4-manager-config\") pod \"lws-controller-manager-8dd669f78-gbkxs\" (UID: \"a21e8431-1dd2-4e8b-9acb-fb420faae4f4\") " pod="openshift-lws-operator/lws-controller-manager-8dd669f78-gbkxs" Sep 16 13:36:45.276440 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.276419 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a21e8431-1dd2-4e8b-9acb-fb420faae4f4-metrics-cert\") pod \"lws-controller-manager-8dd669f78-gbkxs\" (UID: \"a21e8431-1dd2-4e8b-9acb-fb420faae4f4\") " pod="openshift-lws-operator/lws-controller-manager-8dd669f78-gbkxs" Sep 16 13:36:45.276519 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.276466 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a21e8431-1dd2-4e8b-9acb-fb420faae4f4-cert\") pod \"lws-controller-manager-8dd669f78-gbkxs\" (UID: \"a21e8431-1dd2-4e8b-9acb-fb420faae4f4\") " pod="openshift-lws-operator/lws-controller-manager-8dd669f78-gbkxs" Sep 16 13:36:45.283990 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.283966 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf6hb\" (UniqueName: \"kubernetes.io/projected/a21e8431-1dd2-4e8b-9acb-fb420faae4f4-kube-api-access-vf6hb\") pod \"lws-controller-manager-8dd669f78-gbkxs\" (UID: \"a21e8431-1dd2-4e8b-9acb-fb420faae4f4\") " pod="openshift-lws-operator/lws-controller-manager-8dd669f78-gbkxs" Sep 16 13:36:45.404144 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.404063 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8dd669f78-gbkxs" Sep 16 13:36:45.700964 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:45.700943 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8dd669f78-gbkxs"] Sep 16 13:36:45.703681 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:36:45.703654 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21e8431_1dd2_4e8b_9acb_fb420faae4f4.slice/crio-044c5c757512b1462e66df6541356392b0a77eddecf4ac8025a7f10fa2679f5c WatchSource:0}: Error finding container 044c5c757512b1462e66df6541356392b0a77eddecf4ac8025a7f10fa2679f5c: Status 404 returned error can't find the container with id 044c5c757512b1462e66df6541356392b0a77eddecf4ac8025a7f10fa2679f5c Sep 16 13:36:46.092499 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:46.092416 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8dd669f78-gbkxs" event={"ID":"a21e8431-1dd2-4e8b-9acb-fb420faae4f4","Type":"ContainerStarted","Data":"044c5c757512b1462e66df6541356392b0a77eddecf4ac8025a7f10fa2679f5c"} Sep 16 13:36:48.108510 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:48.108408 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8dd669f78-gbkxs" event={"ID":"a21e8431-1dd2-4e8b-9acb-fb420faae4f4","Type":"ContainerStarted","Data":"4ac60fe9b246a6aff943bd5ef728ba1ebbaa298207345b3fb24d3329f68f309e"} Sep 16 13:36:48.108510 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:48.108502 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8dd669f78-gbkxs" Sep 16 13:36:48.130168 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:48.130109 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8dd669f78-gbkxs" podStartSLOduration=1.099049994 podStartE2EDuration="3.130093958s" totalImagesPullingTime="2.031043964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:36:48.128438839 +0000 UTC m=+369.172741825" watchObservedRunningTime="2026-09-16 13:36:48.130093958 +0000 UTC m=+369.174396923" Sep 16 13:36:55.815505 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.815473 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j"] Sep 16 13:36:55.820048 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.820031 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.823461 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.823443 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-lkpgn\"" Sep 16 13:36:55.823575 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.823555 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 16 13:36:55.840506 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.840482 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j"] Sep 16 13:36:55.862552 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.862524 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.862674 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.862577 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.862674 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.862650 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.862759 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.862691 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.862759 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.862724 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.862831 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.862781 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.862831 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.862805 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jkvt\" (UniqueName: \"kubernetes.io/projected/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-kube-api-access-8jkvt\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.862907 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.862852 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.862907 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.862875 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.957592 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.957562 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt"] Sep 16 13:36:55.961016 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.961001 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:55.963371 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.963347 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.963494 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.963406 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.963494 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.963458 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.963494 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.963488 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jkvt\" (UniqueName: \"kubernetes.io/projected/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-kube-api-access-8jkvt\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.963623 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.963538 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.963623 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.963562 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.963702 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.963635 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.963702 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.963676 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.963800 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.963720 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.963926 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.963897 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.964037 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.964000 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.964099 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.964067 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.964158 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.964110 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.964534 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.964513 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.965901 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.965876 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.965974 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.965930 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.984451 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.984433 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt"] Sep 16 13:36:55.985259 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.985235 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:55.986057 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:55.986035 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jkvt\" (UniqueName: \"kubernetes.io/projected/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-kube-api-access-8jkvt\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:56.064635 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.064602 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7726ea1c-381a-4673-9401-fe58c2673873-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.064792 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.064650 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7726ea1c-381a-4673-9401-fe58c2673873-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.064792 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.064674 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7726ea1c-381a-4673-9401-fe58c2673873-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.064792 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.064715 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7726ea1c-381a-4673-9401-fe58c2673873-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.064792 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.064749 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7726ea1c-381a-4673-9401-fe58c2673873-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.064792 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.064784 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-656t9\" (UniqueName: \"kubernetes.io/projected/7726ea1c-381a-4673-9401-fe58c2673873-kube-api-access-656t9\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.064967 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.064800 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7726ea1c-381a-4673-9401-fe58c2673873-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.064967 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.064884 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7726ea1c-381a-4673-9401-fe58c2673873-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.064967 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.064920 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7726ea1c-381a-4673-9401-fe58c2673873-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.129578 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.129507 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:36:56.165988 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.165961 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7726ea1c-381a-4673-9401-fe58c2673873-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.166134 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.165993 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7726ea1c-381a-4673-9401-fe58c2673873-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.166134 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.166044 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7726ea1c-381a-4673-9401-fe58c2673873-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.166134 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.166094 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7726ea1c-381a-4673-9401-fe58c2673873-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.166134 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.166121 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7726ea1c-381a-4673-9401-fe58c2673873-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.166300 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.166154 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7726ea1c-381a-4673-9401-fe58c2673873-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.166300 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.166180 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7726ea1c-381a-4673-9401-fe58c2673873-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.166300 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.166225 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-656t9\" (UniqueName: \"kubernetes.io/projected/7726ea1c-381a-4673-9401-fe58c2673873-kube-api-access-656t9\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.166300 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.166255 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7726ea1c-381a-4673-9401-fe58c2673873-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.166528 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.166450 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7726ea1c-381a-4673-9401-fe58c2673873-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.166528 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.166489 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7726ea1c-381a-4673-9401-fe58c2673873-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.166717 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.166690 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7726ea1c-381a-4673-9401-fe58c2673873-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.166814 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.166721 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7726ea1c-381a-4673-9401-fe58c2673873-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.166920 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.166897 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7726ea1c-381a-4673-9401-fe58c2673873-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.168993 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.168951 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7726ea1c-381a-4673-9401-fe58c2673873-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.169165 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.169143 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7726ea1c-381a-4673-9401-fe58c2673873-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.176033 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.176011 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7726ea1c-381a-4673-9401-fe58c2673873-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.176187 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.176163 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-656t9\" (UniqueName: \"kubernetes.io/projected/7726ea1c-381a-4673-9401-fe58c2673873-kube-api-access-656t9\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c47gnt\" (UID: \"7726ea1c-381a-4673-9401-fe58c2673873\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.285236 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.285196 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:36:56.410689 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.410660 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j"] Sep 16 13:36:56.412204 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:36:56.412179 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2601bc9b_efc4_4543_a67e_67cd5fe7bc1c.slice/crio-36aba6788d46f7a762262e161a779c4872644b323cc896f5168166d28b8164ae WatchSource:0}: Error finding container 36aba6788d46f7a762262e161a779c4872644b323cc896f5168166d28b8164ae: Status 404 returned error can't find the container with id 36aba6788d46f7a762262e161a779c4872644b323cc896f5168166d28b8164ae Sep 16 13:36:56.565879 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:56.565849 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt"] Sep 16 13:36:56.568674 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:36:56.568651 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7726ea1c_381a_4673_9401_fe58c2673873.slice/crio-6efe4fa3ced30bf871804e971c8364bf2dc9fba9f1dbe84a1af28d57a78da4f7 WatchSource:0}: Error finding container 6efe4fa3ced30bf871804e971c8364bf2dc9fba9f1dbe84a1af28d57a78da4f7: Status 404 returned error can't find the container with id 6efe4fa3ced30bf871804e971c8364bf2dc9fba9f1dbe84a1af28d57a78da4f7 Sep 16 13:36:57.151718 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:57.151681 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" event={"ID":"7726ea1c-381a-4673-9401-fe58c2673873","Type":"ContainerStarted","Data":"6efe4fa3ced30bf871804e971c8364bf2dc9fba9f1dbe84a1af28d57a78da4f7"} Sep 16 13:36:57.152838 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:57.152813 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" event={"ID":"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c","Type":"ContainerStarted","Data":"36aba6788d46f7a762262e161a779c4872644b323cc896f5168166d28b8164ae"} Sep 16 13:36:59.117245 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:59.117217 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8dd669f78-gbkxs" Sep 16 13:36:59.322564 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:59.322523 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 16 13:36:59.322669 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:59.322613 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 16 13:36:59.322669 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:59.322655 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 16 13:36:59.329100 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:59.329072 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 16 13:36:59.329188 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:59.329149 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 16 13:36:59.329233 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:36:59.329186 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 16 13:37:00.172898 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:00.172861 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" event={"ID":"7726ea1c-381a-4673-9401-fe58c2673873","Type":"ContainerStarted","Data":"b19f4c4550c91908284fa0e77aa3df49b044d5e70717ac7918eeafbac8626229"} Sep 16 13:37:00.174476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:00.174454 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" event={"ID":"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c","Type":"ContainerStarted","Data":"6284bcf26d1f5e4d33eff4d83b46a0d7d70690c8635d3146553914336d440806"} Sep 16 13:37:00.197713 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:00.197667 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" podStartSLOduration=2.439545664 podStartE2EDuration="5.197653672s" totalImagesPullingTime="2.758108008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:37:00.195506449 +0000 UTC m=+381.239809414" watchObservedRunningTime="2026-09-16 13:37:00.197653672 +0000 UTC m=+381.241956687" Sep 16 13:37:00.226772 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:00.226728 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" podStartSLOduration=2.31873021 podStartE2EDuration="5.226714955s" totalImagesPullingTime="2.907984745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:37:00.225095596 +0000 UTC m=+381.269398587" watchObservedRunningTime="2026-09-16 13:37:00.226714955 +0000 UTC m=+381.271017920" Sep 16 13:37:00.286278 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:00.286251 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:37:00.290908 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:00.290886 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:37:01.130262 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:01.130226 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:37:01.131888 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:01.131866 2562 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Sep 16 13:37:01.131957 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:01.131916 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" podUID="2601bc9b-efc4-4543-a67e-67cd5fe7bc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Sep 16 13:37:01.179924 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:01.179888 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:37:01.180883 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:01.180864 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c47gnt" Sep 16 13:37:01.248677 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:01.248648 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j"] Sep 16 13:37:02.130839 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:02.130806 2562 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Sep 16 13:37:02.131004 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:02.130858 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" podUID="2601bc9b-efc4-4543-a67e-67cd5fe7bc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Sep 16 13:37:03.130760 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:03.130721 2562 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Sep 16 13:37:03.131143 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:03.130790 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" podUID="2601bc9b-efc4-4543-a67e-67cd5fe7bc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Sep 16 13:37:03.188806 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:03.188770 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" podUID="2601bc9b-efc4-4543-a67e-67cd5fe7bc1c" containerName="istio-proxy" containerID="cri-o://6284bcf26d1f5e4d33eff4d83b46a0d7d70690c8635d3146553914336d440806" gracePeriod=30 Sep 16 13:37:08.429740 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.429713 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:37:08.578366 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.578289 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-credential-socket\") pod \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " Sep 16 13:37:08.578366 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.578338 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istio-token\") pod \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " Sep 16 13:37:08.578614 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.578438 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istio-data\") pod \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " Sep 16 13:37:08.578614 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.578460 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istio-envoy\") pod \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " Sep 16 13:37:08.578614 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.578492 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jkvt\" (UniqueName: \"kubernetes.io/projected/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-kube-api-access-8jkvt\") pod \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " Sep 16 13:37:08.578614 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.578541 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-workload-certs\") pod \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " Sep 16 13:37:08.578614 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.578549 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2601bc9b-efc4-4543-a67e-67cd5fe7bc1c" (UID: "2601bc9b-efc4-4543-a67e-67cd5fe7bc1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:37:08.578614 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.578570 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istio-podinfo\") pod \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " Sep 16 13:37:08.578845 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.578663 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istio-data" (OuterVolumeSpecName: "istio-data") pod "2601bc9b-efc4-4543-a67e-67cd5fe7bc1c" (UID: "2601bc9b-efc4-4543-a67e-67cd5fe7bc1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:37:08.578845 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.578713 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-workload-socket\") pod \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " Sep 16 13:37:08.578845 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.578781 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2601bc9b-efc4-4543-a67e-67cd5fe7bc1c" (UID: "2601bc9b-efc4-4543-a67e-67cd5fe7bc1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:37:08.578845 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.578826 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istiod-ca-cert\") pod \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\" (UID: \"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c\") " Sep 16 13:37:08.579078 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.578985 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2601bc9b-efc4-4543-a67e-67cd5fe7bc1c" (UID: "2601bc9b-efc4-4543-a67e-67cd5fe7bc1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:37:08.579167 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.579148 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-workload-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:37:08.579226 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.579174 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-workload-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:37:08.579226 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.579187 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-credential-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:37:08.579226 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.579201 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istio-data\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:37:08.579226 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.579154 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2601bc9b-efc4-4543-a67e-67cd5fe7bc1c" (UID: "2601bc9b-efc4-4543-a67e-67cd5fe7bc1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:37:08.580840 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.580807 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2601bc9b-efc4-4543-a67e-67cd5fe7bc1c" (UID: "2601bc9b-efc4-4543-a67e-67cd5fe7bc1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 13:37:08.580950 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.580841 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istio-token" (OuterVolumeSpecName: "istio-token") pod "2601bc9b-efc4-4543-a67e-67cd5fe7bc1c" (UID: "2601bc9b-efc4-4543-a67e-67cd5fe7bc1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:37:08.581180 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.581159 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2601bc9b-efc4-4543-a67e-67cd5fe7bc1c" (UID: "2601bc9b-efc4-4543-a67e-67cd5fe7bc1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:37:08.581180 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.581165 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-kube-api-access-8jkvt" (OuterVolumeSpecName: "kube-api-access-8jkvt") pod "2601bc9b-efc4-4543-a67e-67cd5fe7bc1c" (UID: "2601bc9b-efc4-4543-a67e-67cd5fe7bc1c"). InnerVolumeSpecName "kube-api-access-8jkvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:37:08.680524 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.680493 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istio-envoy\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:37:08.680524 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.680518 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jkvt\" (UniqueName: \"kubernetes.io/projected/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-kube-api-access-8jkvt\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:37:08.680524 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.680529 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istio-podinfo\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:37:08.680739 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.680537 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istiod-ca-cert\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:37:08.680739 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.680544 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c-istio-token\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:37:08.870525 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.870448 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-st8pk"] Sep 16 13:37:08.870822 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.870811 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2601bc9b-efc4-4543-a67e-67cd5fe7bc1c" containerName="istio-proxy" Sep 16 13:37:08.870868 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.870825 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2601bc9b-efc4-4543-a67e-67cd5fe7bc1c" containerName="istio-proxy" Sep 16 13:37:08.870901 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.870893 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2601bc9b-efc4-4543-a67e-67cd5fe7bc1c" containerName="istio-proxy" Sep 16 13:37:08.873114 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.873096 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-st8pk" Sep 16 13:37:08.882350 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.882331 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-hkwkh\"" Sep 16 13:37:08.882527 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.882512 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 16 13:37:08.913299 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.913277 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-st8pk"] Sep 16 13:37:08.915079 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.915062 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8f55z"] Sep 16 13:37:08.917359 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.917344 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8f55z" Sep 16 13:37:08.928073 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.928051 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-gffr2\"" Sep 16 13:37:08.928174 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.928147 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 16 13:37:08.949142 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.949020 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8f55z"] Sep 16 13:37:08.982729 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.982706 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrsm8\" (UniqueName: \"kubernetes.io/projected/071707f9-2c28-4395-a424-cdae5211a023-kube-api-access-qrsm8\") pod \"llmisvc-controller-manager-777c6bd7d5-8f55z\" (UID: \"071707f9-2c28-4395-a424-cdae5211a023\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8f55z" Sep 16 13:37:08.982874 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.982734 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c0255ad-b36c-497e-a92b-d65807812dd8-cert\") pod \"kserve-controller-manager-d6f4b65b8-st8pk\" (UID: \"0c0255ad-b36c-497e-a92b-d65807812dd8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-st8pk" Sep 16 13:37:08.982874 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.982756 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp54s\" (UniqueName: \"kubernetes.io/projected/0c0255ad-b36c-497e-a92b-d65807812dd8-kube-api-access-xp54s\") pod \"kserve-controller-manager-d6f4b65b8-st8pk\" (UID: \"0c0255ad-b36c-497e-a92b-d65807812dd8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-st8pk" Sep 16 13:37:08.982980 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:08.982882 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/071707f9-2c28-4395-a424-cdae5211a023-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8f55z\" (UID: \"071707f9-2c28-4395-a424-cdae5211a023\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8f55z" Sep 16 13:37:09.083660 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:09.083626 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrsm8\" (UniqueName: \"kubernetes.io/projected/071707f9-2c28-4395-a424-cdae5211a023-kube-api-access-qrsm8\") pod \"llmisvc-controller-manager-777c6bd7d5-8f55z\" (UID: \"071707f9-2c28-4395-a424-cdae5211a023\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8f55z" Sep 16 13:37:09.083660 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:09.083660 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c0255ad-b36c-497e-a92b-d65807812dd8-cert\") pod \"kserve-controller-manager-d6f4b65b8-st8pk\" (UID: \"0c0255ad-b36c-497e-a92b-d65807812dd8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-st8pk" Sep 16 13:37:09.083847 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:09.083688 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp54s\" (UniqueName: \"kubernetes.io/projected/0c0255ad-b36c-497e-a92b-d65807812dd8-kube-api-access-xp54s\") pod \"kserve-controller-manager-d6f4b65b8-st8pk\" (UID: \"0c0255ad-b36c-497e-a92b-d65807812dd8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-st8pk" Sep 16 13:37:09.083847 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:09.083746 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/071707f9-2c28-4395-a424-cdae5211a023-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8f55z\" (UID: \"071707f9-2c28-4395-a424-cdae5211a023\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8f55z" Sep 16 13:37:09.086146 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:09.086110 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c0255ad-b36c-497e-a92b-d65807812dd8-cert\") pod \"kserve-controller-manager-d6f4b65b8-st8pk\" (UID: \"0c0255ad-b36c-497e-a92b-d65807812dd8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-st8pk" Sep 16 13:37:09.086244 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:09.086233 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/071707f9-2c28-4395-a424-cdae5211a023-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8f55z\" (UID: \"071707f9-2c28-4395-a424-cdae5211a023\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8f55z" Sep 16 13:37:09.094921 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:09.094896 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrsm8\" (UniqueName: \"kubernetes.io/projected/071707f9-2c28-4395-a424-cdae5211a023-kube-api-access-qrsm8\") pod \"llmisvc-controller-manager-777c6bd7d5-8f55z\" (UID: \"071707f9-2c28-4395-a424-cdae5211a023\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8f55z" Sep 16 13:37:09.095651 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:09.095637 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp54s\" (UniqueName: \"kubernetes.io/projected/0c0255ad-b36c-497e-a92b-d65807812dd8-kube-api-access-xp54s\") pod \"kserve-controller-manager-d6f4b65b8-st8pk\" (UID: \"0c0255ad-b36c-497e-a92b-d65807812dd8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-st8pk" Sep 16 13:37:09.183251 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:09.183220 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-st8pk" Sep 16 13:37:09.225994 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:09.225970 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8f55z" Sep 16 13:37:09.226196 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:09.226025 2562 generic.go:356] "Generic (PLEG): container finished" podID="2601bc9b-efc4-4543-a67e-67cd5fe7bc1c" containerID="6284bcf26d1f5e4d33eff4d83b46a0d7d70690c8635d3146553914336d440806" exitCode=0 Sep 16 13:37:09.226196 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:09.226129 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" Sep 16 13:37:09.226196 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:09.226129 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" event={"ID":"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c","Type":"ContainerDied","Data":"6284bcf26d1f5e4d33eff4d83b46a0d7d70690c8635d3146553914336d440806"} Sep 16 13:37:09.226354 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:09.226250 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j" event={"ID":"2601bc9b-efc4-4543-a67e-67cd5fe7bc1c","Type":"ContainerDied","Data":"36aba6788d46f7a762262e161a779c4872644b323cc896f5168166d28b8164ae"} Sep 16 13:37:09.226354 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:09.226275 2562 scope.go:117] "RemoveContainer" containerID="6284bcf26d1f5e4d33eff4d83b46a0d7d70690c8635d3146553914336d440806" Sep 16 13:37:09.242081 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:09.242063 2562 scope.go:117] "RemoveContainer" containerID="6284bcf26d1f5e4d33eff4d83b46a0d7d70690c8635d3146553914336d440806" Sep 16 13:37:09.243209 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:37:09.242747 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6284bcf26d1f5e4d33eff4d83b46a0d7d70690c8635d3146553914336d440806\": container with ID starting with 6284bcf26d1f5e4d33eff4d83b46a0d7d70690c8635d3146553914336d440806 not found: ID does not exist" containerID="6284bcf26d1f5e4d33eff4d83b46a0d7d70690c8635d3146553914336d440806" Sep 16 13:37:09.243209 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:09.242864 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6284bcf26d1f5e4d33eff4d83b46a0d7d70690c8635d3146553914336d440806"} err="failed to get container status \"6284bcf26d1f5e4d33eff4d83b46a0d7d70690c8635d3146553914336d440806\": rpc error: code = NotFound desc = could not find container \"6284bcf26d1f5e4d33eff4d83b46a0d7d70690c8635d3146553914336d440806\": container with ID starting with 6284bcf26d1f5e4d33eff4d83b46a0d7d70690c8635d3146553914336d440806 not found: ID does not exist" Sep 16 13:37:09.280752 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:09.280714 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j"] Sep 16 13:37:09.300413 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:09.300351 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-2vl8j"] Sep 16 13:37:09.412854 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:09.412820 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2601bc9b-efc4-4543-a67e-67cd5fe7bc1c" path="/var/lib/kubelet/pods/2601bc9b-efc4-4543-a67e-67cd5fe7bc1c/volumes" Sep 16 13:37:09.498162 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:09.498092 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-st8pk"] Sep 16 13:37:09.500731 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:37:09.500698 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0255ad_b36c_497e_a92b_d65807812dd8.slice/crio-ec426d0b6242a65aaccac523ad58033678a6d29bb4815a5b49a3d9cfd48150c1 WatchSource:0}: Error finding container ec426d0b6242a65aaccac523ad58033678a6d29bb4815a5b49a3d9cfd48150c1: Status 404 returned error can't find the container with id ec426d0b6242a65aaccac523ad58033678a6d29bb4815a5b49a3d9cfd48150c1 Sep 16 13:37:09.538771 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:09.538744 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8f55z"] Sep 16 13:37:09.542067 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:37:09.542039 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod071707f9_2c28_4395_a424_cdae5211a023.slice/crio-596c563e4b7d9fbaf4c707e8a711f59907d332d1039abe821aa25c88c246ff1e WatchSource:0}: Error finding container 596c563e4b7d9fbaf4c707e8a711f59907d332d1039abe821aa25c88c246ff1e: Status 404 returned error can't find the container with id 596c563e4b7d9fbaf4c707e8a711f59907d332d1039abe821aa25c88c246ff1e Sep 16 13:37:10.238010 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:10.237941 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8f55z" event={"ID":"071707f9-2c28-4395-a424-cdae5211a023","Type":"ContainerStarted","Data":"596c563e4b7d9fbaf4c707e8a711f59907d332d1039abe821aa25c88c246ff1e"} Sep 16 13:37:10.239829 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:10.239793 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-st8pk" event={"ID":"0c0255ad-b36c-497e-a92b-d65807812dd8","Type":"ContainerStarted","Data":"ec426d0b6242a65aaccac523ad58033678a6d29bb4815a5b49a3d9cfd48150c1"} Sep 16 13:37:13.261128 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:13.261087 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8f55z" event={"ID":"071707f9-2c28-4395-a424-cdae5211a023","Type":"ContainerStarted","Data":"d2ed3f4ce4650d099cbc9c7daf122f7726e06637ea4e7aef1c82c3ec6501f24b"} Sep 16 13:37:13.261632 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:13.261219 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8f55z" Sep 16 13:37:13.262724 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:13.262702 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-st8pk" event={"ID":"0c0255ad-b36c-497e-a92b-d65807812dd8","Type":"ContainerStarted","Data":"794df1f20efc224d4b4d51349b92469797db0f1ac0d8e36232301254b983daab"} Sep 16 13:37:13.262831 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:13.262748 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-st8pk" Sep 16 13:37:13.311672 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:13.311583 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8f55z" podStartSLOduration=1.757354168 podStartE2EDuration="5.311567344s" totalImagesPullingTime="3.554213176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:37:13.28487709 +0000 UTC m=+394.329180074" watchObservedRunningTime="2026-09-16 13:37:13.311567344 +0000 UTC m=+394.355870312" Sep 16 13:37:13.312656 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:13.312628 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-st8pk" podStartSLOduration=1.7562962199999999 podStartE2EDuration="5.312616712s" totalImagesPullingTime="3.556320492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:37:13.311494011 +0000 UTC m=+394.355796971" watchObservedRunningTime="2026-09-16 13:37:13.312616712 +0000 UTC m=+394.356919677" Sep 16 13:37:44.270540 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:44.270507 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8f55z" Sep 16 13:37:44.273789 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:37:44.273768 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-st8pk" Sep 16 13:38:03.534764 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.534685 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j"] Sep 16 13:38:03.537245 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.537230 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j" Sep 16 13:38:03.540187 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.540163 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 13:38:03.541767 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.541749 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 13:38:03.541866 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.541773 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-75dbt\"" Sep 16 13:38:03.547822 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.547799 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j"] Sep 16 13:38:03.626891 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.626859 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85"] Sep 16 13:38:03.629776 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.629759 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85" Sep 16 13:38:03.632748 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.632720 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3785644-49b3-4e7e-9f00-b7e769eb3819-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j\" (UID: \"e3785644-49b3-4e7e-9f00-b7e769eb3819\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j" Sep 16 13:38:03.632833 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.632766 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q8gj\" (UniqueName: \"kubernetes.io/projected/e3785644-49b3-4e7e-9f00-b7e769eb3819-kube-api-access-7q8gj\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j\" (UID: \"e3785644-49b3-4e7e-9f00-b7e769eb3819\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j" Sep 16 13:38:03.632833 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.632826 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3785644-49b3-4e7e-9f00-b7e769eb3819-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j\" (UID: \"e3785644-49b3-4e7e-9f00-b7e769eb3819\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j" Sep 16 13:38:03.640154 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.640133 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85"] Sep 16 13:38:03.733511 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.733471 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73812c10-2355-4db9-8a01-bd13e8f1a349-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85\" (UID: \"73812c10-2355-4db9-8a01-bd13e8f1a349\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85" Sep 16 13:38:03.733688 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.733558 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rsjg\" (UniqueName: \"kubernetes.io/projected/73812c10-2355-4db9-8a01-bd13e8f1a349-kube-api-access-9rsjg\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85\" (UID: \"73812c10-2355-4db9-8a01-bd13e8f1a349\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85" Sep 16 13:38:03.733688 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.733592 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73812c10-2355-4db9-8a01-bd13e8f1a349-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85\" (UID: \"73812c10-2355-4db9-8a01-bd13e8f1a349\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85" Sep 16 13:38:03.733781 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.733686 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3785644-49b3-4e7e-9f00-b7e769eb3819-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j\" (UID: \"e3785644-49b3-4e7e-9f00-b7e769eb3819\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j" Sep 16 13:38:03.733781 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.733713 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q8gj\" (UniqueName: \"kubernetes.io/projected/e3785644-49b3-4e7e-9f00-b7e769eb3819-kube-api-access-7q8gj\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j\" (UID: \"e3785644-49b3-4e7e-9f00-b7e769eb3819\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j" Sep 16 13:38:03.733781 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.733736 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3785644-49b3-4e7e-9f00-b7e769eb3819-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j\" (UID: \"e3785644-49b3-4e7e-9f00-b7e769eb3819\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j" Sep 16 13:38:03.734088 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.734068 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3785644-49b3-4e7e-9f00-b7e769eb3819-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j\" (UID: \"e3785644-49b3-4e7e-9f00-b7e769eb3819\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j" Sep 16 13:38:03.734170 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.734074 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3785644-49b3-4e7e-9f00-b7e769eb3819-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j\" (UID: \"e3785644-49b3-4e7e-9f00-b7e769eb3819\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j" Sep 16 13:38:03.735604 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.735587 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq"] Sep 16 13:38:03.738104 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.738091 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq" Sep 16 13:38:03.742704 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.742680 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q8gj\" (UniqueName: \"kubernetes.io/projected/e3785644-49b3-4e7e-9f00-b7e769eb3819-kube-api-access-7q8gj\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j\" (UID: \"e3785644-49b3-4e7e-9f00-b7e769eb3819\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j" Sep 16 13:38:03.750006 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.749984 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq"] Sep 16 13:38:03.834318 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.834225 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/061af4aa-3375-4103-a843-73b3c60d6bd6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq\" (UID: \"061af4aa-3375-4103-a843-73b3c60d6bd6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq" Sep 16 13:38:03.834318 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.834268 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rsjg\" (UniqueName: \"kubernetes.io/projected/73812c10-2355-4db9-8a01-bd13e8f1a349-kube-api-access-9rsjg\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85\" (UID: \"73812c10-2355-4db9-8a01-bd13e8f1a349\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85" Sep 16 13:38:03.834318 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.834283 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/061af4aa-3375-4103-a843-73b3c60d6bd6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq\" (UID: \"061af4aa-3375-4103-a843-73b3c60d6bd6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq" Sep 16 13:38:03.834318 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.834304 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73812c10-2355-4db9-8a01-bd13e8f1a349-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85\" (UID: \"73812c10-2355-4db9-8a01-bd13e8f1a349\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85" Sep 16 13:38:03.834678 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.834435 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thptq\" (UniqueName: \"kubernetes.io/projected/061af4aa-3375-4103-a843-73b3c60d6bd6-kube-api-access-thptq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq\" (UID: \"061af4aa-3375-4103-a843-73b3c60d6bd6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq" Sep 16 13:38:03.834678 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.834484 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73812c10-2355-4db9-8a01-bd13e8f1a349-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85\" (UID: \"73812c10-2355-4db9-8a01-bd13e8f1a349\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85" Sep 16 13:38:03.834751 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.834687 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73812c10-2355-4db9-8a01-bd13e8f1a349-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85\" (UID: \"73812c10-2355-4db9-8a01-bd13e8f1a349\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85" Sep 16 13:38:03.834751 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.834736 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73812c10-2355-4db9-8a01-bd13e8f1a349-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85\" (UID: \"73812c10-2355-4db9-8a01-bd13e8f1a349\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85" Sep 16 13:38:03.846142 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.846101 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rsjg\" (UniqueName: \"kubernetes.io/projected/73812c10-2355-4db9-8a01-bd13e8f1a349-kube-api-access-9rsjg\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85\" (UID: \"73812c10-2355-4db9-8a01-bd13e8f1a349\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85" Sep 16 13:38:03.846285 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.846238 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j" Sep 16 13:38:03.855560 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.855538 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq"] Sep 16 13:38:03.858146 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.858127 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq" Sep 16 13:38:03.886814 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.886581 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq"] Sep 16 13:38:03.935193 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.935161 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c90be9d-297e-42ce-b699-245403c9dedd-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq\" (UID: \"1c90be9d-297e-42ce-b699-245403c9dedd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq" Sep 16 13:38:03.935406 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.935224 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c90be9d-297e-42ce-b699-245403c9dedd-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq\" (UID: \"1c90be9d-297e-42ce-b699-245403c9dedd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq" Sep 16 13:38:03.935406 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.935272 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thptq\" (UniqueName: \"kubernetes.io/projected/061af4aa-3375-4103-a843-73b3c60d6bd6-kube-api-access-thptq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq\" (UID: \"061af4aa-3375-4103-a843-73b3c60d6bd6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq" Sep 16 13:38:03.935406 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.935313 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmzk9\" (UniqueName: \"kubernetes.io/projected/1c90be9d-297e-42ce-b699-245403c9dedd-kube-api-access-pmzk9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq\" (UID: \"1c90be9d-297e-42ce-b699-245403c9dedd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq" Sep 16 13:38:03.935406 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.935360 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/061af4aa-3375-4103-a843-73b3c60d6bd6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq\" (UID: \"061af4aa-3375-4103-a843-73b3c60d6bd6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq" Sep 16 13:38:03.935406 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.935405 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/061af4aa-3375-4103-a843-73b3c60d6bd6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq\" (UID: \"061af4aa-3375-4103-a843-73b3c60d6bd6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq" Sep 16 13:38:03.935781 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.935761 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/061af4aa-3375-4103-a843-73b3c60d6bd6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq\" (UID: \"061af4aa-3375-4103-a843-73b3c60d6bd6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq" Sep 16 13:38:03.935816 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.935774 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/061af4aa-3375-4103-a843-73b3c60d6bd6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq\" (UID: \"061af4aa-3375-4103-a843-73b3c60d6bd6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq" Sep 16 13:38:03.939084 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.939064 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85" Sep 16 13:38:03.946583 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:03.946560 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thptq\" (UniqueName: \"kubernetes.io/projected/061af4aa-3375-4103-a843-73b3c60d6bd6-kube-api-access-thptq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq\" (UID: \"061af4aa-3375-4103-a843-73b3c60d6bd6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq" Sep 16 13:38:04.035962 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:04.035934 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c90be9d-297e-42ce-b699-245403c9dedd-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq\" (UID: \"1c90be9d-297e-42ce-b699-245403c9dedd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq" Sep 16 13:38:04.036119 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:04.035986 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c90be9d-297e-42ce-b699-245403c9dedd-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq\" (UID: \"1c90be9d-297e-42ce-b699-245403c9dedd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq" Sep 16 13:38:04.036119 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:04.036036 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmzk9\" (UniqueName: \"kubernetes.io/projected/1c90be9d-297e-42ce-b699-245403c9dedd-kube-api-access-pmzk9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq\" (UID: \"1c90be9d-297e-42ce-b699-245403c9dedd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq" Sep 16 13:38:04.036358 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:04.036337 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c90be9d-297e-42ce-b699-245403c9dedd-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq\" (UID: \"1c90be9d-297e-42ce-b699-245403c9dedd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq" Sep 16 13:38:04.036414 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:04.036344 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c90be9d-297e-42ce-b699-245403c9dedd-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq\" (UID: \"1c90be9d-297e-42ce-b699-245403c9dedd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq" Sep 16 13:38:04.046316 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:04.046290 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmzk9\" (UniqueName: \"kubernetes.io/projected/1c90be9d-297e-42ce-b699-245403c9dedd-kube-api-access-pmzk9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq\" (UID: \"1c90be9d-297e-42ce-b699-245403c9dedd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq" Sep 16 13:38:04.056193 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:04.056167 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq" Sep 16 13:38:04.126710 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:04.126682 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j"] Sep 16 13:38:04.129232 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:38:04.129206 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3785644_49b3_4e7e_9f00_b7e769eb3819.slice/crio-677410ef5b48f40549501d8f60c7bb270d7bfb168357e5b5603c0079a166caa2 WatchSource:0}: Error finding container 677410ef5b48f40549501d8f60c7bb270d7bfb168357e5b5603c0079a166caa2: Status 404 returned error can't find the container with id 677410ef5b48f40549501d8f60c7bb270d7bfb168357e5b5603c0079a166caa2 Sep 16 13:38:04.179971 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:04.179946 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq" Sep 16 13:38:04.218401 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:04.218356 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85"] Sep 16 13:38:04.219724 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:38:04.219699 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73812c10_2355_4db9_8a01_bd13e8f1a349.slice/crio-e029537e4e6a1c405a5b28138ee849fe438535699c39885fbe43dba598d95ba5 WatchSource:0}: Error finding container e029537e4e6a1c405a5b28138ee849fe438535699c39885fbe43dba598d95ba5: Status 404 returned error can't find the container with id e029537e4e6a1c405a5b28138ee849fe438535699c39885fbe43dba598d95ba5 Sep 16 13:38:04.333152 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:04.333122 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq"] Sep 16 13:38:04.335074 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:38:04.335049 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061af4aa_3375_4103_a843_73b3c60d6bd6.slice/crio-65c0f55f5536f968e529bf110dd132363a410224be2ebc8dca36f9b2bb955b40 WatchSource:0}: Error finding container 65c0f55f5536f968e529bf110dd132363a410224be2ebc8dca36f9b2bb955b40: Status 404 returned error can't find the container with id 65c0f55f5536f968e529bf110dd132363a410224be2ebc8dca36f9b2bb955b40 Sep 16 13:38:04.463635 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:04.463610 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq"] Sep 16 13:38:04.465646 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:38:04.465596 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c90be9d_297e_42ce_b699_245403c9dedd.slice/crio-6b4621b490885538887c84056f4c006373c93b016436a31568157840c52c82c7 WatchSource:0}: Error finding container 6b4621b490885538887c84056f4c006373c93b016436a31568157840c52c82c7: Status 404 returned error can't find the container with id 6b4621b490885538887c84056f4c006373c93b016436a31568157840c52c82c7 Sep 16 13:38:04.509919 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:04.509885 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq" event={"ID":"1c90be9d-297e-42ce-b699-245403c9dedd","Type":"ContainerStarted","Data":"6b4621b490885538887c84056f4c006373c93b016436a31568157840c52c82c7"} Sep 16 13:38:04.511531 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:04.511496 2562 generic.go:356] "Generic (PLEG): container finished" podID="061af4aa-3375-4103-a843-73b3c60d6bd6" containerID="9d028f4f7ebaebe697a2dcbf3f9cc95072db2e2cdb6ece736d4ad8662be63883" exitCode=0 Sep 16 13:38:04.511678 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:04.511582 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq" event={"ID":"061af4aa-3375-4103-a843-73b3c60d6bd6","Type":"ContainerDied","Data":"9d028f4f7ebaebe697a2dcbf3f9cc95072db2e2cdb6ece736d4ad8662be63883"} Sep 16 13:38:04.511678 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:04.511632 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq" event={"ID":"061af4aa-3375-4103-a843-73b3c60d6bd6","Type":"ContainerStarted","Data":"65c0f55f5536f968e529bf110dd132363a410224be2ebc8dca36f9b2bb955b40"} Sep 16 13:38:04.517112 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:04.513929 2562 generic.go:356] "Generic (PLEG): container finished" podID="73812c10-2355-4db9-8a01-bd13e8f1a349" containerID="55c28683f6b914609362a928c34243e4e0ec4ab234dedddcf9b388e0150f5d60" exitCode=0 Sep 16 13:38:04.517112 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:04.514045 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85" event={"ID":"73812c10-2355-4db9-8a01-bd13e8f1a349","Type":"ContainerDied","Data":"55c28683f6b914609362a928c34243e4e0ec4ab234dedddcf9b388e0150f5d60"} Sep 16 13:38:04.517112 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:04.514136 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85" event={"ID":"73812c10-2355-4db9-8a01-bd13e8f1a349","Type":"ContainerStarted","Data":"e029537e4e6a1c405a5b28138ee849fe438535699c39885fbe43dba598d95ba5"} Sep 16 13:38:04.522513 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:04.522472 2562 generic.go:356] "Generic (PLEG): container finished" podID="e3785644-49b3-4e7e-9f00-b7e769eb3819" containerID="2f07cc2e0368f79324c6a4dca701989cb98f321bce5e085c858a51417db785d0" exitCode=0 Sep 16 13:38:04.522592 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:04.522513 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j" event={"ID":"e3785644-49b3-4e7e-9f00-b7e769eb3819","Type":"ContainerDied","Data":"2f07cc2e0368f79324c6a4dca701989cb98f321bce5e085c858a51417db785d0"} Sep 16 13:38:04.522592 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:04.522549 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j" event={"ID":"e3785644-49b3-4e7e-9f00-b7e769eb3819","Type":"ContainerStarted","Data":"677410ef5b48f40549501d8f60c7bb270d7bfb168357e5b5603c0079a166caa2"} Sep 16 13:38:05.529702 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:05.529670 2562 generic.go:356] "Generic (PLEG): container finished" podID="1c90be9d-297e-42ce-b699-245403c9dedd" containerID="e9a18ada5b42df0c88dbcc09738e1beebfc2257c160449e11ec97052e70e71de" exitCode=0 Sep 16 13:38:05.530143 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:05.529748 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq" event={"ID":"1c90be9d-297e-42ce-b699-245403c9dedd","Type":"ContainerDied","Data":"e9a18ada5b42df0c88dbcc09738e1beebfc2257c160449e11ec97052e70e71de"} Sep 16 13:38:05.531813 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:05.531794 2562 generic.go:356] "Generic (PLEG): container finished" podID="061af4aa-3375-4103-a843-73b3c60d6bd6" containerID="dcc3dadc5b89e9fdc19ad18f3271087597c0c3faf10b73e2d60543b17ec5dc8d" exitCode=0 Sep 16 13:38:05.531905 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:05.531874 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq" event={"ID":"061af4aa-3375-4103-a843-73b3c60d6bd6","Type":"ContainerDied","Data":"dcc3dadc5b89e9fdc19ad18f3271087597c0c3faf10b73e2d60543b17ec5dc8d"} Sep 16 13:38:05.533968 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:05.533945 2562 generic.go:356] "Generic (PLEG): container finished" podID="73812c10-2355-4db9-8a01-bd13e8f1a349" containerID="bb86f4fd938caf7a04724758a9dccc7d7485607bb33581449145e4158223b1fc" exitCode=0 Sep 16 13:38:05.534068 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:05.534016 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85" event={"ID":"73812c10-2355-4db9-8a01-bd13e8f1a349","Type":"ContainerDied","Data":"bb86f4fd938caf7a04724758a9dccc7d7485607bb33581449145e4158223b1fc"} Sep 16 13:38:05.536089 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:05.536070 2562 generic.go:356] "Generic (PLEG): container finished" podID="e3785644-49b3-4e7e-9f00-b7e769eb3819" containerID="a0800ec0e896b4d0a988518800f26b948ad01d438a986e00e612e05fe74f10bc" exitCode=0 Sep 16 13:38:05.536187 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:05.536130 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j" event={"ID":"e3785644-49b3-4e7e-9f00-b7e769eb3819","Type":"ContainerDied","Data":"a0800ec0e896b4d0a988518800f26b948ad01d438a986e00e612e05fe74f10bc"} Sep 16 13:38:06.546228 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:06.546180 2562 generic.go:356] "Generic (PLEG): container finished" podID="73812c10-2355-4db9-8a01-bd13e8f1a349" containerID="9510291ca2bbcfad036123020cb3bb88fcb96d43cad47ed490afaf78e314aec5" exitCode=0 Sep 16 13:38:06.546228 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:06.546224 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85" event={"ID":"73812c10-2355-4db9-8a01-bd13e8f1a349","Type":"ContainerDied","Data":"9510291ca2bbcfad036123020cb3bb88fcb96d43cad47ed490afaf78e314aec5"} Sep 16 13:38:06.548444 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:06.548412 2562 generic.go:356] "Generic (PLEG): container finished" podID="e3785644-49b3-4e7e-9f00-b7e769eb3819" containerID="f32fb83a6bdbe4056c8fb62835062f9a1ee45efa3c053ef07fc86d9681133b09" exitCode=0 Sep 16 13:38:06.548554 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:06.548487 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j" event={"ID":"e3785644-49b3-4e7e-9f00-b7e769eb3819","Type":"ContainerDied","Data":"f32fb83a6bdbe4056c8fb62835062f9a1ee45efa3c053ef07fc86d9681133b09"} Sep 16 13:38:06.550274 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:06.550251 2562 generic.go:356] "Generic (PLEG): container finished" podID="1c90be9d-297e-42ce-b699-245403c9dedd" containerID="cab919813b58a2dea6f0d214491824614eafa42d008fa0f2d115c22aa829a609" exitCode=0 Sep 16 13:38:06.550413 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:06.550333 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq" event={"ID":"1c90be9d-297e-42ce-b699-245403c9dedd","Type":"ContainerDied","Data":"cab919813b58a2dea6f0d214491824614eafa42d008fa0f2d115c22aa829a609"} Sep 16 13:38:06.552770 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:06.552750 2562 generic.go:356] "Generic (PLEG): container finished" podID="061af4aa-3375-4103-a843-73b3c60d6bd6" containerID="06f0538c537cb7deaeb973c4876d558a86f630a69133464732415cb4f9cdd028" exitCode=0 Sep 16 13:38:06.552866 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:06.552835 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq" event={"ID":"061af4aa-3375-4103-a843-73b3c60d6bd6","Type":"ContainerDied","Data":"06f0538c537cb7deaeb973c4876d558a86f630a69133464732415cb4f9cdd028"} Sep 16 13:38:07.560544 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.560514 2562 generic.go:356] "Generic (PLEG): container finished" podID="1c90be9d-297e-42ce-b699-245403c9dedd" containerID="d3e15ccd64ca057ee049f3905f2adaad9f154e932da2ab3e8e4e6b5aa13a0df8" exitCode=0 Sep 16 13:38:07.560960 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.560594 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq" event={"ID":"1c90be9d-297e-42ce-b699-245403c9dedd","Type":"ContainerDied","Data":"d3e15ccd64ca057ee049f3905f2adaad9f154e932da2ab3e8e4e6b5aa13a0df8"} Sep 16 13:38:07.698419 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.698397 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq" Sep 16 13:38:07.754643 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.754620 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j" Sep 16 13:38:07.756659 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.756643 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85" Sep 16 13:38:07.771687 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.771667 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/061af4aa-3375-4103-a843-73b3c60d6bd6-bundle\") pod \"061af4aa-3375-4103-a843-73b3c60d6bd6\" (UID: \"061af4aa-3375-4103-a843-73b3c60d6bd6\") " Sep 16 13:38:07.771758 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.771698 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/061af4aa-3375-4103-a843-73b3c60d6bd6-util\") pod \"061af4aa-3375-4103-a843-73b3c60d6bd6\" (UID: \"061af4aa-3375-4103-a843-73b3c60d6bd6\") " Sep 16 13:38:07.771815 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.771802 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thptq\" (UniqueName: \"kubernetes.io/projected/061af4aa-3375-4103-a843-73b3c60d6bd6-kube-api-access-thptq\") pod \"061af4aa-3375-4103-a843-73b3c60d6bd6\" (UID: \"061af4aa-3375-4103-a843-73b3c60d6bd6\") " Sep 16 13:38:07.772080 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.772059 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061af4aa-3375-4103-a843-73b3c60d6bd6-bundle" (OuterVolumeSpecName: "bundle") pod "061af4aa-3375-4103-a843-73b3c60d6bd6" (UID: "061af4aa-3375-4103-a843-73b3c60d6bd6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:38:07.773925 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.773897 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061af4aa-3375-4103-a843-73b3c60d6bd6-kube-api-access-thptq" (OuterVolumeSpecName: "kube-api-access-thptq") pod "061af4aa-3375-4103-a843-73b3c60d6bd6" (UID: "061af4aa-3375-4103-a843-73b3c60d6bd6"). InnerVolumeSpecName "kube-api-access-thptq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:38:07.777443 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.777363 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061af4aa-3375-4103-a843-73b3c60d6bd6-util" (OuterVolumeSpecName: "util") pod "061af4aa-3375-4103-a843-73b3c60d6bd6" (UID: "061af4aa-3375-4103-a843-73b3c60d6bd6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:38:07.872955 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.872875 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rsjg\" (UniqueName: \"kubernetes.io/projected/73812c10-2355-4db9-8a01-bd13e8f1a349-kube-api-access-9rsjg\") pod \"73812c10-2355-4db9-8a01-bd13e8f1a349\" (UID: \"73812c10-2355-4db9-8a01-bd13e8f1a349\") " Sep 16 13:38:07.872955 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.872916 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73812c10-2355-4db9-8a01-bd13e8f1a349-bundle\") pod \"73812c10-2355-4db9-8a01-bd13e8f1a349\" (UID: \"73812c10-2355-4db9-8a01-bd13e8f1a349\") " Sep 16 13:38:07.872955 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.872957 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3785644-49b3-4e7e-9f00-b7e769eb3819-bundle\") pod \"e3785644-49b3-4e7e-9f00-b7e769eb3819\" (UID: \"e3785644-49b3-4e7e-9f00-b7e769eb3819\") " Sep 16 13:38:07.873193 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.872993 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3785644-49b3-4e7e-9f00-b7e769eb3819-util\") pod \"e3785644-49b3-4e7e-9f00-b7e769eb3819\" (UID: \"e3785644-49b3-4e7e-9f00-b7e769eb3819\") " Sep 16 13:38:07.873193 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.873006 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73812c10-2355-4db9-8a01-bd13e8f1a349-util\") pod \"73812c10-2355-4db9-8a01-bd13e8f1a349\" (UID: \"73812c10-2355-4db9-8a01-bd13e8f1a349\") " Sep 16 13:38:07.873193 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.873028 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7q8gj\" (UniqueName: \"kubernetes.io/projected/e3785644-49b3-4e7e-9f00-b7e769eb3819-kube-api-access-7q8gj\") pod \"e3785644-49b3-4e7e-9f00-b7e769eb3819\" (UID: \"e3785644-49b3-4e7e-9f00-b7e769eb3819\") " Sep 16 13:38:07.873343 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.873329 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thptq\" (UniqueName: \"kubernetes.io/projected/061af4aa-3375-4103-a843-73b3c60d6bd6-kube-api-access-thptq\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:38:07.873431 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.873351 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/061af4aa-3375-4103-a843-73b3c60d6bd6-bundle\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:38:07.873431 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.873365 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/061af4aa-3375-4103-a843-73b3c60d6bd6-util\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:38:07.873570 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.873542 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3785644-49b3-4e7e-9f00-b7e769eb3819-bundle" (OuterVolumeSpecName: "bundle") pod "e3785644-49b3-4e7e-9f00-b7e769eb3819" (UID: "e3785644-49b3-4e7e-9f00-b7e769eb3819"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:38:07.873670 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.873647 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73812c10-2355-4db9-8a01-bd13e8f1a349-bundle" (OuterVolumeSpecName: "bundle") pod "73812c10-2355-4db9-8a01-bd13e8f1a349" (UID: "73812c10-2355-4db9-8a01-bd13e8f1a349"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:38:07.875275 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.875252 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73812c10-2355-4db9-8a01-bd13e8f1a349-kube-api-access-9rsjg" (OuterVolumeSpecName: "kube-api-access-9rsjg") pod "73812c10-2355-4db9-8a01-bd13e8f1a349" (UID: "73812c10-2355-4db9-8a01-bd13e8f1a349"). InnerVolumeSpecName "kube-api-access-9rsjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:38:07.875357 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.875274 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3785644-49b3-4e7e-9f00-b7e769eb3819-kube-api-access-7q8gj" (OuterVolumeSpecName: "kube-api-access-7q8gj") pod "e3785644-49b3-4e7e-9f00-b7e769eb3819" (UID: "e3785644-49b3-4e7e-9f00-b7e769eb3819"). InnerVolumeSpecName "kube-api-access-7q8gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:38:07.881366 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.881325 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3785644-49b3-4e7e-9f00-b7e769eb3819-util" (OuterVolumeSpecName: "util") pod "e3785644-49b3-4e7e-9f00-b7e769eb3819" (UID: "e3785644-49b3-4e7e-9f00-b7e769eb3819"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:38:07.881876 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.881855 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73812c10-2355-4db9-8a01-bd13e8f1a349-util" (OuterVolumeSpecName: "util") pod "73812c10-2355-4db9-8a01-bd13e8f1a349" (UID: "73812c10-2355-4db9-8a01-bd13e8f1a349"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:38:07.974244 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.974196 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3785644-49b3-4e7e-9f00-b7e769eb3819-util\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:38:07.974244 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.974238 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73812c10-2355-4db9-8a01-bd13e8f1a349-util\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:38:07.974244 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.974251 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q8gj\" (UniqueName: \"kubernetes.io/projected/e3785644-49b3-4e7e-9f00-b7e769eb3819-kube-api-access-7q8gj\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:38:07.974513 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.974264 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rsjg\" (UniqueName: \"kubernetes.io/projected/73812c10-2355-4db9-8a01-bd13e8f1a349-kube-api-access-9rsjg\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:38:07.974513 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.974276 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73812c10-2355-4db9-8a01-bd13e8f1a349-bundle\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:38:07.974513 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:07.974289 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3785644-49b3-4e7e-9f00-b7e769eb3819-bundle\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:38:08.566962 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:08.566931 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq" Sep 16 13:38:08.566962 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:08.566958 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69whzq" event={"ID":"061af4aa-3375-4103-a843-73b3c60d6bd6","Type":"ContainerDied","Data":"65c0f55f5536f968e529bf110dd132363a410224be2ebc8dca36f9b2bb955b40"} Sep 16 13:38:08.567412 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:08.567002 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65c0f55f5536f968e529bf110dd132363a410224be2ebc8dca36f9b2bb955b40" Sep 16 13:38:08.568864 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:08.568846 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85" Sep 16 13:38:08.568984 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:08.568841 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csqc85" event={"ID":"73812c10-2355-4db9-8a01-bd13e8f1a349","Type":"ContainerDied","Data":"e029537e4e6a1c405a5b28138ee849fe438535699c39885fbe43dba598d95ba5"} Sep 16 13:38:08.568984 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:08.568930 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e029537e4e6a1c405a5b28138ee849fe438535699c39885fbe43dba598d95ba5" Sep 16 13:38:08.570871 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:08.570855 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j" Sep 16 13:38:08.570871 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:08.570846 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xpz4j" event={"ID":"e3785644-49b3-4e7e-9f00-b7e769eb3819","Type":"ContainerDied","Data":"677410ef5b48f40549501d8f60c7bb270d7bfb168357e5b5603c0079a166caa2"} Sep 16 13:38:08.570996 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:08.570900 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="677410ef5b48f40549501d8f60c7bb270d7bfb168357e5b5603c0079a166caa2" Sep 16 13:38:08.702342 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:08.702320 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq" Sep 16 13:38:08.780184 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:08.780147 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmzk9\" (UniqueName: \"kubernetes.io/projected/1c90be9d-297e-42ce-b699-245403c9dedd-kube-api-access-pmzk9\") pod \"1c90be9d-297e-42ce-b699-245403c9dedd\" (UID: \"1c90be9d-297e-42ce-b699-245403c9dedd\") " Sep 16 13:38:08.780409 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:08.780193 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c90be9d-297e-42ce-b699-245403c9dedd-bundle\") pod \"1c90be9d-297e-42ce-b699-245403c9dedd\" (UID: \"1c90be9d-297e-42ce-b699-245403c9dedd\") " Sep 16 13:38:08.780409 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:08.780277 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c90be9d-297e-42ce-b699-245403c9dedd-util\") pod \"1c90be9d-297e-42ce-b699-245403c9dedd\" (UID: \"1c90be9d-297e-42ce-b699-245403c9dedd\") " Sep 16 13:38:08.780878 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:08.780809 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c90be9d-297e-42ce-b699-245403c9dedd-bundle" (OuterVolumeSpecName: "bundle") pod "1c90be9d-297e-42ce-b699-245403c9dedd" (UID: "1c90be9d-297e-42ce-b699-245403c9dedd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:38:08.782517 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:08.782490 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c90be9d-297e-42ce-b699-245403c9dedd-kube-api-access-pmzk9" (OuterVolumeSpecName: "kube-api-access-pmzk9") pod "1c90be9d-297e-42ce-b699-245403c9dedd" (UID: "1c90be9d-297e-42ce-b699-245403c9dedd"). InnerVolumeSpecName "kube-api-access-pmzk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:38:08.786145 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:08.786127 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c90be9d-297e-42ce-b699-245403c9dedd-util" (OuterVolumeSpecName: "util") pod "1c90be9d-297e-42ce-b699-245403c9dedd" (UID: "1c90be9d-297e-42ce-b699-245403c9dedd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:38:08.881785 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:08.881703 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c90be9d-297e-42ce-b699-245403c9dedd-util\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:38:08.881785 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:08.881728 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmzk9\" (UniqueName: \"kubernetes.io/projected/1c90be9d-297e-42ce-b699-245403c9dedd-kube-api-access-pmzk9\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:38:08.881785 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:08.881739 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c90be9d-297e-42ce-b699-245403c9dedd-bundle\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:38:09.578018 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:09.577987 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq" Sep 16 13:38:09.578491 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:09.578023 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8v6jq" event={"ID":"1c90be9d-297e-42ce-b699-245403c9dedd","Type":"ContainerDied","Data":"6b4621b490885538887c84056f4c006373c93b016436a31568157840c52c82c7"} Sep 16 13:38:09.578491 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:09.578066 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b4621b490885538887c84056f4c006373c93b016436a31568157840c52c82c7" Sep 16 13:38:23.620994 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.620959 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-whlj7"] Sep 16 13:38:23.621476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621298 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="061af4aa-3375-4103-a843-73b3c60d6bd6" containerName="util" Sep 16 13:38:23.621476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621307 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="061af4aa-3375-4103-a843-73b3c60d6bd6" containerName="util" Sep 16 13:38:23.621476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621321 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c90be9d-297e-42ce-b699-245403c9dedd" containerName="extract" Sep 16 13:38:23.621476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621326 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c90be9d-297e-42ce-b699-245403c9dedd" containerName="extract" Sep 16 13:38:23.621476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621332 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3785644-49b3-4e7e-9f00-b7e769eb3819" containerName="pull" Sep 16 13:38:23.621476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621338 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3785644-49b3-4e7e-9f00-b7e769eb3819" containerName="pull" Sep 16 13:38:23.621476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621344 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3785644-49b3-4e7e-9f00-b7e769eb3819" containerName="util" Sep 16 13:38:23.621476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621349 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3785644-49b3-4e7e-9f00-b7e769eb3819" containerName="util" Sep 16 13:38:23.621476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621356 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73812c10-2355-4db9-8a01-bd13e8f1a349" containerName="extract" Sep 16 13:38:23.621476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621361 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="73812c10-2355-4db9-8a01-bd13e8f1a349" containerName="extract" Sep 16 13:38:23.621476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621372 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73812c10-2355-4db9-8a01-bd13e8f1a349" containerName="pull" Sep 16 13:38:23.621476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621393 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="73812c10-2355-4db9-8a01-bd13e8f1a349" containerName="pull" Sep 16 13:38:23.621476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621399 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c90be9d-297e-42ce-b699-245403c9dedd" containerName="util" Sep 16 13:38:23.621476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621404 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c90be9d-297e-42ce-b699-245403c9dedd" containerName="util" Sep 16 13:38:23.621476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621410 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="061af4aa-3375-4103-a843-73b3c60d6bd6" containerName="pull" Sep 16 13:38:23.621476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621415 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="061af4aa-3375-4103-a843-73b3c60d6bd6" containerName="pull" Sep 16 13:38:23.621476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621420 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c90be9d-297e-42ce-b699-245403c9dedd" containerName="pull" Sep 16 13:38:23.621476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621424 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c90be9d-297e-42ce-b699-245403c9dedd" containerName="pull" Sep 16 13:38:23.621476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621430 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="061af4aa-3375-4103-a843-73b3c60d6bd6" containerName="extract" Sep 16 13:38:23.621476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621435 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="061af4aa-3375-4103-a843-73b3c60d6bd6" containerName="extract" Sep 16 13:38:23.621476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621442 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3785644-49b3-4e7e-9f00-b7e769eb3819" containerName="extract" Sep 16 13:38:23.621476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621447 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3785644-49b3-4e7e-9f00-b7e769eb3819" containerName="extract" Sep 16 13:38:23.621476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621452 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73812c10-2355-4db9-8a01-bd13e8f1a349" containerName="util" Sep 16 13:38:23.621476 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621457 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="73812c10-2355-4db9-8a01-bd13e8f1a349" containerName="util" Sep 16 13:38:23.622299 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621508 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="061af4aa-3375-4103-a843-73b3c60d6bd6" containerName="extract" Sep 16 13:38:23.622299 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621517 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c90be9d-297e-42ce-b699-245403c9dedd" containerName="extract" Sep 16 13:38:23.622299 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621522 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3785644-49b3-4e7e-9f00-b7e769eb3819" containerName="extract" Sep 16 13:38:23.622299 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.621528 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="73812c10-2355-4db9-8a01-bd13e8f1a349" containerName="extract" Sep 16 13:38:23.625781 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.625758 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-whlj7" Sep 16 13:38:23.630824 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.630768 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 16 13:38:23.630959 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.630921 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-f5bx2\"" Sep 16 13:38:23.631040 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.630986 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 16 13:38:23.631040 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.631016 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 16 13:38:23.675578 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.675552 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-whlj7"] Sep 16 13:38:23.704962 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.704934 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9kbz\" (UniqueName: \"kubernetes.io/projected/75d365c2-8196-4d1d-a77f-fedf7d5b0531-kube-api-access-h9kbz\") pod \"dns-operator-controller-manager-7b84cd7c9c-whlj7\" (UID: \"75d365c2-8196-4d1d-a77f-fedf7d5b0531\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-whlj7" Sep 16 13:38:23.806128 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.806071 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9kbz\" (UniqueName: \"kubernetes.io/projected/75d365c2-8196-4d1d-a77f-fedf7d5b0531-kube-api-access-h9kbz\") pod \"dns-operator-controller-manager-7b84cd7c9c-whlj7\" (UID: \"75d365c2-8196-4d1d-a77f-fedf7d5b0531\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-whlj7" Sep 16 13:38:23.835471 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.834531 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9kbz\" (UniqueName: \"kubernetes.io/projected/75d365c2-8196-4d1d-a77f-fedf7d5b0531-kube-api-access-h9kbz\") pod \"dns-operator-controller-manager-7b84cd7c9c-whlj7\" (UID: \"75d365c2-8196-4d1d-a77f-fedf7d5b0531\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-whlj7" Sep 16 13:38:23.935275 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:23.935230 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-whlj7" Sep 16 13:38:24.219633 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:24.219578 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-whlj7"] Sep 16 13:38:24.221907 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:38:24.221881 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d365c2_8196_4d1d_a77f_fedf7d5b0531.slice/crio-8f4592c3046d650f641ca8a2ee3ba480b2ced9a2da54aae50dd3be3888301f62 WatchSource:0}: Error finding container 8f4592c3046d650f641ca8a2ee3ba480b2ced9a2da54aae50dd3be3888301f62: Status 404 returned error can't find the container with id 8f4592c3046d650f641ca8a2ee3ba480b2ced9a2da54aae50dd3be3888301f62 Sep 16 13:38:24.663805 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:24.663767 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-whlj7" event={"ID":"75d365c2-8196-4d1d-a77f-fedf7d5b0531","Type":"ContainerStarted","Data":"8f4592c3046d650f641ca8a2ee3ba480b2ced9a2da54aae50dd3be3888301f62"} Sep 16 13:38:25.508960 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:25.508909 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qvlcj"] Sep 16 13:38:25.512613 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:25.512584 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qvlcj" Sep 16 13:38:25.516682 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:25.516654 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-k2nzf\"" Sep 16 13:38:25.531428 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:25.531401 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qvlcj"] Sep 16 13:38:25.621829 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:25.621790 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vsxw\" (UniqueName: \"kubernetes.io/projected/80189c69-39a4-4374-82ca-802ee54fe122-kube-api-access-4vsxw\") pod \"limitador-operator-controller-manager-74495c69f8-qvlcj\" (UID: \"80189c69-39a4-4374-82ca-802ee54fe122\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qvlcj" Sep 16 13:38:25.722617 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:25.722580 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vsxw\" (UniqueName: \"kubernetes.io/projected/80189c69-39a4-4374-82ca-802ee54fe122-kube-api-access-4vsxw\") pod \"limitador-operator-controller-manager-74495c69f8-qvlcj\" (UID: \"80189c69-39a4-4374-82ca-802ee54fe122\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qvlcj" Sep 16 13:38:25.742925 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:25.742893 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vsxw\" (UniqueName: \"kubernetes.io/projected/80189c69-39a4-4374-82ca-802ee54fe122-kube-api-access-4vsxw\") pod \"limitador-operator-controller-manager-74495c69f8-qvlcj\" (UID: \"80189c69-39a4-4374-82ca-802ee54fe122\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qvlcj" Sep 16 13:38:25.825714 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:25.825625 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qvlcj" Sep 16 13:38:26.677132 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:26.677101 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-whlj7" event={"ID":"75d365c2-8196-4d1d-a77f-fedf7d5b0531","Type":"ContainerStarted","Data":"5492f6fb2a14619abc4c34f7c81197c6e2f8e61124da91a7f5dd0a5fe7b11146"} Sep 16 13:38:26.677301 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:26.677167 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-whlj7" Sep 16 13:38:26.703919 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:26.703869 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-whlj7" podStartSLOduration=1.338493261 podStartE2EDuration="3.703849408s" totalImagesPullingTime="2.365356147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:38:26.701257081 +0000 UTC m=+467.745560048" watchObservedRunningTime="2026-09-16 13:38:26.703849408 +0000 UTC m=+467.748152373" Sep 16 13:38:26.793580 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:26.793556 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qvlcj"] Sep 16 13:38:26.795315 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:38:26.795288 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80189c69_39a4_4374_82ca_802ee54fe122.slice/crio-f625518e5c5301c8cacab17694f03ea70c56d95a44e9f06287271e011c28d073 WatchSource:0}: Error finding container f625518e5c5301c8cacab17694f03ea70c56d95a44e9f06287271e011c28d073: Status 404 returned error can't find the container with id f625518e5c5301c8cacab17694f03ea70c56d95a44e9f06287271e011c28d073 Sep 16 13:38:27.684406 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:27.684357 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qvlcj" event={"ID":"80189c69-39a4-4374-82ca-802ee54fe122","Type":"ContainerStarted","Data":"f625518e5c5301c8cacab17694f03ea70c56d95a44e9f06287271e011c28d073"} Sep 16 13:38:28.697308 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:28.697269 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qvlcj" event={"ID":"80189c69-39a4-4374-82ca-802ee54fe122","Type":"ContainerStarted","Data":"60f01c61427f41320301d8b2bb1790e4b772b188094e16cd3cd6490ecc9d1f8a"} Sep 16 13:38:28.697824 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:28.697430 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qvlcj" Sep 16 13:38:28.741152 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:28.741098 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qvlcj" podStartSLOduration=2.482667168 podStartE2EDuration="3.741082623s" totalImagesPullingTime="1.258415455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:38:28.738969366 +0000 UTC m=+469.783272343" watchObservedRunningTime="2026-09-16 13:38:28.741082623 +0000 UTC m=+469.785385580" Sep 16 13:38:37.687154 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:37.687122 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-whlj7" Sep 16 13:38:38.172738 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:38.172709 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qvlcj"] Sep 16 13:38:38.172946 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:38.172902 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qvlcj" podUID="80189c69-39a4-4374-82ca-802ee54fe122" containerName="manager" containerID="cri-o://60f01c61427f41320301d8b2bb1790e4b772b188094e16cd3cd6490ecc9d1f8a" gracePeriod=2 Sep 16 13:38:38.175403 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:38.175352 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qvlcj" Sep 16 13:38:38.183514 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:38.183493 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qvlcj"] Sep 16 13:38:38.197886 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:38:38.197848 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qvlcj\" is forbidden: User \"system:node:ip-10-0-139-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-23.ec2.internal' and this object" podUID="80189c69-39a4-4374-82ca-802ee54fe122" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qvlcj" Sep 16 13:38:38.251907 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:38.251878 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2d6q"] Sep 16 13:38:38.252258 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:38.252246 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80189c69-39a4-4374-82ca-802ee54fe122" containerName="manager" Sep 16 13:38:38.252297 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:38.252260 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="80189c69-39a4-4374-82ca-802ee54fe122" containerName="manager" Sep 16 13:38:38.252327 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:38.252318 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="80189c69-39a4-4374-82ca-802ee54fe122" containerName="manager" Sep 16 13:38:38.254208 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:38.254193 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2d6q" Sep 16 13:38:38.282630 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:38.282605 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2d6q"] Sep 16 13:38:38.303262 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:38:38.303220 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qvlcj\" is forbidden: User \"system:node:ip-10-0-139-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-23.ec2.internal' and this object" podUID="80189c69-39a4-4374-82ca-802ee54fe122" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qvlcj" Sep 16 13:38:38.421484 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:38.421462 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qvlcj" Sep 16 13:38:38.424482 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:38:38.424435 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qvlcj\" is forbidden: User \"system:node:ip-10-0-139-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-23.ec2.internal' and this object" podUID="80189c69-39a4-4374-82ca-802ee54fe122" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qvlcj" Sep 16 13:38:38.433472 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:38.433451 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8khn7\" (UniqueName: \"kubernetes.io/projected/35a2ca89-4444-401b-a953-1541396b6be0-kube-api-access-8khn7\") pod \"limitador-operator-controller-manager-74495c69f8-r2d6q\" (UID: \"35a2ca89-4444-401b-a953-1541396b6be0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2d6q" Sep 16 13:38:38.534839 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:38.534812 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vsxw\" (UniqueName: \"kubernetes.io/projected/80189c69-39a4-4374-82ca-802ee54fe122-kube-api-access-4vsxw\") pod \"80189c69-39a4-4374-82ca-802ee54fe122\" (UID: \"80189c69-39a4-4374-82ca-802ee54fe122\") " Sep 16 13:38:38.535011 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:38.534988 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8khn7\" (UniqueName: \"kubernetes.io/projected/35a2ca89-4444-401b-a953-1541396b6be0-kube-api-access-8khn7\") pod \"limitador-operator-controller-manager-74495c69f8-r2d6q\" (UID: \"35a2ca89-4444-401b-a953-1541396b6be0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2d6q" Sep 16 13:38:38.537181 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:38.537157 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80189c69-39a4-4374-82ca-802ee54fe122-kube-api-access-4vsxw" (OuterVolumeSpecName: "kube-api-access-4vsxw") pod "80189c69-39a4-4374-82ca-802ee54fe122" (UID: "80189c69-39a4-4374-82ca-802ee54fe122"). InnerVolumeSpecName "kube-api-access-4vsxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:38:38.559961 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:38.559939 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8khn7\" (UniqueName: \"kubernetes.io/projected/35a2ca89-4444-401b-a953-1541396b6be0-kube-api-access-8khn7\") pod \"limitador-operator-controller-manager-74495c69f8-r2d6q\" (UID: \"35a2ca89-4444-401b-a953-1541396b6be0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2d6q" Sep 16 13:38:38.565709 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:38.565688 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2d6q" Sep 16 13:38:38.636284 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:38.636255 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vsxw\" (UniqueName: \"kubernetes.io/projected/80189c69-39a4-4374-82ca-802ee54fe122-kube-api-access-4vsxw\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:38:38.750866 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:38.750785 2562 generic.go:356] "Generic (PLEG): container finished" podID="80189c69-39a4-4374-82ca-802ee54fe122" containerID="60f01c61427f41320301d8b2bb1790e4b772b188094e16cd3cd6490ecc9d1f8a" exitCode=0 Sep 16 13:38:38.750866 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:38.750825 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qvlcj" Sep 16 13:38:38.751310 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:38.750873 2562 scope.go:117] "RemoveContainer" containerID="60f01c61427f41320301d8b2bb1790e4b772b188094e16cd3cd6490ecc9d1f8a" Sep 16 13:38:38.755735 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:38:38.755706 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qvlcj\" is forbidden: User \"system:node:ip-10-0-139-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-23.ec2.internal' and this object" podUID="80189c69-39a4-4374-82ca-802ee54fe122" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qvlcj" Sep 16 13:38:38.760578 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:38.760561 2562 scope.go:117] "RemoveContainer" containerID="60f01c61427f41320301d8b2bb1790e4b772b188094e16cd3cd6490ecc9d1f8a" Sep 16 13:38:38.760806 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:38:38.760791 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f01c61427f41320301d8b2bb1790e4b772b188094e16cd3cd6490ecc9d1f8a\": container with ID starting with 60f01c61427f41320301d8b2bb1790e4b772b188094e16cd3cd6490ecc9d1f8a not found: ID does not exist" containerID="60f01c61427f41320301d8b2bb1790e4b772b188094e16cd3cd6490ecc9d1f8a" Sep 16 13:38:38.760859 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:38.760813 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f01c61427f41320301d8b2bb1790e4b772b188094e16cd3cd6490ecc9d1f8a"} err="failed to get container status \"60f01c61427f41320301d8b2bb1790e4b772b188094e16cd3cd6490ecc9d1f8a\": rpc error: code = NotFound desc = could not find container \"60f01c61427f41320301d8b2bb1790e4b772b188094e16cd3cd6490ecc9d1f8a\": container with ID starting with 60f01c61427f41320301d8b2bb1790e4b772b188094e16cd3cd6490ecc9d1f8a not found: ID does not exist" Sep 16 13:38:38.762528 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:38:38.762509 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qvlcj\" is forbidden: User \"system:node:ip-10-0-139-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-23.ec2.internal' and this object" podUID="80189c69-39a4-4374-82ca-802ee54fe122" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qvlcj" Sep 16 13:38:38.873759 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:38:38.873728 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a2ca89_4444_401b_a953_1541396b6be0.slice/crio-829603857d28c8e31123c2f91ea58e5a3e82b1f2434f504b52cb52d1acad1789 WatchSource:0}: Error finding container 829603857d28c8e31123c2f91ea58e5a3e82b1f2434f504b52cb52d1acad1789: Status 404 returned error can't find the container with id 829603857d28c8e31123c2f91ea58e5a3e82b1f2434f504b52cb52d1acad1789 Sep 16 13:38:38.873856 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:38.873787 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2d6q"] Sep 16 13:38:39.413140 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:39.413113 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80189c69-39a4-4374-82ca-802ee54fe122" path="/var/lib/kubelet/pods/80189c69-39a4-4374-82ca-802ee54fe122/volumes" Sep 16 13:38:39.413574 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:38:39.413552 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qvlcj\" is forbidden: User \"system:node:ip-10-0-139-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-23.ec2.internal' and this object" podUID="80189c69-39a4-4374-82ca-802ee54fe122" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qvlcj" Sep 16 13:38:39.758025 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:39.757996 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2d6q" event={"ID":"35a2ca89-4444-401b-a953-1541396b6be0","Type":"ContainerStarted","Data":"6b426824ecc51b4b039b776084d01bfe2015428e9a7f0613f51f7b999a6cc8e3"} Sep 16 13:38:39.758416 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:39.758042 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2d6q" event={"ID":"35a2ca89-4444-401b-a953-1541396b6be0","Type":"ContainerStarted","Data":"829603857d28c8e31123c2f91ea58e5a3e82b1f2434f504b52cb52d1acad1789"} Sep 16 13:38:39.758416 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:39.758128 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2d6q" Sep 16 13:38:39.780862 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:39.780819 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2d6q" podStartSLOduration=1.780803205 podStartE2EDuration="1.780803205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:38:39.780109824 +0000 UTC m=+480.824412811" watchObservedRunningTime="2026-09-16 13:38:39.780803205 +0000 UTC m=+480.825106173" Sep 16 13:38:50.764957 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:38:50.764926 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-r2d6q" Sep 16 13:39:09.464437 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.464407 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8"] Sep 16 13:39:09.468306 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.468286 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.471547 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.471530 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-54bw5\"" Sep 16 13:39:09.482577 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.482552 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8"] Sep 16 13:39:09.572773 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.572744 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fb052b-58e9-441f-8139-f34dcca5ffbb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.572926 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.572783 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgrd5\" (UniqueName: \"kubernetes.io/projected/22fb052b-58e9-441f-8139-f34dcca5ffbb-kube-api-access-jgrd5\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.572926 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.572818 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fb052b-58e9-441f-8139-f34dcca5ffbb-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.572926 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.572850 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fb052b-58e9-441f-8139-f34dcca5ffbb-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.572926 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.572888 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fb052b-58e9-441f-8139-f34dcca5ffbb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.572926 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.572910 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fb052b-58e9-441f-8139-f34dcca5ffbb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.572926 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.572924 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22fb052b-58e9-441f-8139-f34dcca5ffbb-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.573120 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.572964 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fb052b-58e9-441f-8139-f34dcca5ffbb-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.573120 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.572989 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fb052b-58e9-441f-8139-f34dcca5ffbb-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.673775 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.673746 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fb052b-58e9-441f-8139-f34dcca5ffbb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.673949 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.673784 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgrd5\" (UniqueName: \"kubernetes.io/projected/22fb052b-58e9-441f-8139-f34dcca5ffbb-kube-api-access-jgrd5\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.673949 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.673815 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fb052b-58e9-441f-8139-f34dcca5ffbb-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.673949 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.673840 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fb052b-58e9-441f-8139-f34dcca5ffbb-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.673949 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.673879 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fb052b-58e9-441f-8139-f34dcca5ffbb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.673949 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.673906 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fb052b-58e9-441f-8139-f34dcca5ffbb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.673949 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.673930 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22fb052b-58e9-441f-8139-f34dcca5ffbb-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.674275 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.673977 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fb052b-58e9-441f-8139-f34dcca5ffbb-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.674275 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.674007 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fb052b-58e9-441f-8139-f34dcca5ffbb-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.674407 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.674318 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fb052b-58e9-441f-8139-f34dcca5ffbb-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.674407 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.674348 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fb052b-58e9-441f-8139-f34dcca5ffbb-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.674517 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.674456 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fb052b-58e9-441f-8139-f34dcca5ffbb-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.674580 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.674514 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fb052b-58e9-441f-8139-f34dcca5ffbb-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.674742 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.674719 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fb052b-58e9-441f-8139-f34dcca5ffbb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.676308 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.676288 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fb052b-58e9-441f-8139-f34dcca5ffbb-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.676561 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.676545 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fb052b-58e9-441f-8139-f34dcca5ffbb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.682599 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.682576 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgrd5\" (UniqueName: \"kubernetes.io/projected/22fb052b-58e9-441f-8139-f34dcca5ffbb-kube-api-access-jgrd5\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.682803 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.682785 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22fb052b-58e9-441f-8139-f34dcca5ffbb-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tszg8\" (UID: \"22fb052b-58e9-441f-8139-f34dcca5ffbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:09.778198 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:09.778126 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:10.055463 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:10.055443 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8"] Sep 16 13:39:10.057061 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:39:10.057025 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22fb052b_58e9_441f_8139_f34dcca5ffbb.slice/crio-42bb393e34516c7be30eaccb4e0ff4c43cf0014d3ee0ab6ab57366c0fff53a87 WatchSource:0}: Error finding container 42bb393e34516c7be30eaccb4e0ff4c43cf0014d3ee0ab6ab57366c0fff53a87: Status 404 returned error can't find the container with id 42bb393e34516c7be30eaccb4e0ff4c43cf0014d3ee0ab6ab57366c0fff53a87 Sep 16 13:39:10.059146 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:10.059111 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 16 13:39:10.059245 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:10.059228 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 16 13:39:10.059294 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:10.059278 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 16 13:39:10.920051 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:10.920021 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" event={"ID":"22fb052b-58e9-441f-8139-f34dcca5ffbb","Type":"ContainerStarted","Data":"b0c307d514eaa25e98dbb43d43b8980d79cd73685b56de854458f86230b47b48"} Sep 16 13:39:10.920434 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:10.920080 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" event={"ID":"22fb052b-58e9-441f-8139-f34dcca5ffbb","Type":"ContainerStarted","Data":"42bb393e34516c7be30eaccb4e0ff4c43cf0014d3ee0ab6ab57366c0fff53a87"} Sep 16 13:39:10.947898 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:10.947855 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" podStartSLOduration=1.947840012 podStartE2EDuration="1.947840012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:39:10.946081663 +0000 UTC m=+511.990384665" watchObservedRunningTime="2026-09-16 13:39:10.947840012 +0000 UTC m=+511.992142978" Sep 16 13:39:11.778561 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:11.778524 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:11.783252 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:11.783227 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:11.925451 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:11.925418 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:11.926421 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:11.926400 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:39:13.051058 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:13.051027 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-6wgzf"] Sep 16 13:39:13.054594 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:13.054574 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6wgzf" Sep 16 13:39:13.057789 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:13.057767 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xfzjj\"" Sep 16 13:39:13.064996 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:13.064969 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6wgzf"] Sep 16 13:39:13.103593 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:13.103568 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv29m\" (UniqueName: \"kubernetes.io/projected/08cd9ef6-fcea-4b77-90bd-cd4887246328-kube-api-access-nv29m\") pod \"authorino-59c974bd48-6wgzf\" (UID: \"08cd9ef6-fcea-4b77-90bd-cd4887246328\") " pod="kuadrant-system/authorino-59c974bd48-6wgzf" Sep 16 13:39:13.204177 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:13.204152 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv29m\" (UniqueName: \"kubernetes.io/projected/08cd9ef6-fcea-4b77-90bd-cd4887246328-kube-api-access-nv29m\") pod \"authorino-59c974bd48-6wgzf\" (UID: \"08cd9ef6-fcea-4b77-90bd-cd4887246328\") " pod="kuadrant-system/authorino-59c974bd48-6wgzf" Sep 16 13:39:13.213112 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:13.213090 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv29m\" (UniqueName: \"kubernetes.io/projected/08cd9ef6-fcea-4b77-90bd-cd4887246328-kube-api-access-nv29m\") pod \"authorino-59c974bd48-6wgzf\" (UID: \"08cd9ef6-fcea-4b77-90bd-cd4887246328\") " pod="kuadrant-system/authorino-59c974bd48-6wgzf" Sep 16 13:39:13.364550 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:13.364463 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6wgzf" Sep 16 13:39:13.634681 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:13.634654 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6wgzf"] Sep 16 13:39:13.636054 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:39:13.636029 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08cd9ef6_fcea_4b77_90bd_cd4887246328.slice/crio-4b3ae6f4f8cd9fca2f83c59186660ee550b82cdd8caa7068024bd9545231fc25 WatchSource:0}: Error finding container 4b3ae6f4f8cd9fca2f83c59186660ee550b82cdd8caa7068024bd9545231fc25: Status 404 returned error can't find the container with id 4b3ae6f4f8cd9fca2f83c59186660ee550b82cdd8caa7068024bd9545231fc25 Sep 16 13:39:13.936848 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:13.936818 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6wgzf" event={"ID":"08cd9ef6-fcea-4b77-90bd-cd4887246328","Type":"ContainerStarted","Data":"4b3ae6f4f8cd9fca2f83c59186660ee550b82cdd8caa7068024bd9545231fc25"} Sep 16 13:39:15.949540 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:15.949499 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6wgzf" event={"ID":"08cd9ef6-fcea-4b77-90bd-cd4887246328","Type":"ContainerStarted","Data":"e683ac1cb7f768abffdb40360df94285cdc5dbc4484bbdc22dc83186a508800f"} Sep 16 13:39:15.967068 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:15.967024 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-6wgzf" podStartSLOduration=1.1078882349999999 podStartE2EDuration="2.967009234s" totalImagesPullingTime="1.859120999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:39:15.965813957 +0000 UTC m=+517.010116920" watchObservedRunningTime="2026-09-16 13:39:15.967009234 +0000 UTC m=+517.011312200" Sep 16 13:39:48.591555 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:48.591471 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6wgzf"] Sep 16 13:39:48.592553 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:48.591711 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-6wgzf" podUID="08cd9ef6-fcea-4b77-90bd-cd4887246328" containerName="authorino" containerID="cri-o://e683ac1cb7f768abffdb40360df94285cdc5dbc4484bbdc22dc83186a508800f" gracePeriod=30 Sep 16 13:39:48.833230 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:48.833205 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6wgzf" Sep 16 13:39:48.986800 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:48.986772 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nv29m\" (UniqueName: \"kubernetes.io/projected/08cd9ef6-fcea-4b77-90bd-cd4887246328-kube-api-access-nv29m\") pod \"08cd9ef6-fcea-4b77-90bd-cd4887246328\" (UID: \"08cd9ef6-fcea-4b77-90bd-cd4887246328\") " Sep 16 13:39:48.988893 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:48.988870 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08cd9ef6-fcea-4b77-90bd-cd4887246328-kube-api-access-nv29m" (OuterVolumeSpecName: "kube-api-access-nv29m") pod "08cd9ef6-fcea-4b77-90bd-cd4887246328" (UID: "08cd9ef6-fcea-4b77-90bd-cd4887246328"). InnerVolumeSpecName "kube-api-access-nv29m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:39:49.087939 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:49.087919 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv29m\" (UniqueName: \"kubernetes.io/projected/08cd9ef6-fcea-4b77-90bd-cd4887246328-kube-api-access-nv29m\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:39:49.123942 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:49.123914 2562 generic.go:356] "Generic (PLEG): container finished" podID="08cd9ef6-fcea-4b77-90bd-cd4887246328" containerID="e683ac1cb7f768abffdb40360df94285cdc5dbc4484bbdc22dc83186a508800f" exitCode=0 Sep 16 13:39:49.124052 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:49.123965 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6wgzf" Sep 16 13:39:49.124052 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:49.123983 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6wgzf" event={"ID":"08cd9ef6-fcea-4b77-90bd-cd4887246328","Type":"ContainerDied","Data":"e683ac1cb7f768abffdb40360df94285cdc5dbc4484bbdc22dc83186a508800f"} Sep 16 13:39:49.124052 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:49.124026 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6wgzf" event={"ID":"08cd9ef6-fcea-4b77-90bd-cd4887246328","Type":"ContainerDied","Data":"4b3ae6f4f8cd9fca2f83c59186660ee550b82cdd8caa7068024bd9545231fc25"} Sep 16 13:39:49.124052 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:49.124048 2562 scope.go:117] "RemoveContainer" containerID="e683ac1cb7f768abffdb40360df94285cdc5dbc4484bbdc22dc83186a508800f" Sep 16 13:39:49.136763 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:49.136343 2562 scope.go:117] "RemoveContainer" containerID="e683ac1cb7f768abffdb40360df94285cdc5dbc4484bbdc22dc83186a508800f" Sep 16 13:39:49.137034 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:39:49.137003 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e683ac1cb7f768abffdb40360df94285cdc5dbc4484bbdc22dc83186a508800f\": container with ID starting with e683ac1cb7f768abffdb40360df94285cdc5dbc4484bbdc22dc83186a508800f not found: ID does not exist" containerID="e683ac1cb7f768abffdb40360df94285cdc5dbc4484bbdc22dc83186a508800f" Sep 16 13:39:49.137104 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:49.137070 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e683ac1cb7f768abffdb40360df94285cdc5dbc4484bbdc22dc83186a508800f"} err="failed to get container status \"e683ac1cb7f768abffdb40360df94285cdc5dbc4484bbdc22dc83186a508800f\": rpc error: code = NotFound desc = could not find container \"e683ac1cb7f768abffdb40360df94285cdc5dbc4484bbdc22dc83186a508800f\": container with ID starting with e683ac1cb7f768abffdb40360df94285cdc5dbc4484bbdc22dc83186a508800f not found: ID does not exist" Sep 16 13:39:49.152866 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:49.152846 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6wgzf"] Sep 16 13:39:49.161354 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:49.161336 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6wgzf"] Sep 16 13:39:49.413733 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:39:49.413706 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08cd9ef6-fcea-4b77-90bd-cd4887246328" path="/var/lib/kubelet/pods/08cd9ef6-fcea-4b77-90bd-cd4887246328/volumes" Sep 16 13:40:30.776543 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:30.776510 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s"] Sep 16 13:40:30.776997 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:30.776891 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08cd9ef6-fcea-4b77-90bd-cd4887246328" containerName="authorino" Sep 16 13:40:30.776997 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:30.776902 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="08cd9ef6-fcea-4b77-90bd-cd4887246328" containerName="authorino" Sep 16 13:40:30.776997 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:30.776983 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="08cd9ef6-fcea-4b77-90bd-cd4887246328" containerName="authorino" Sep 16 13:40:30.780167 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:30.780146 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s" Sep 16 13:40:30.784659 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:30.784638 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 13:40:30.784885 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:30.784701 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 13:40:30.784885 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:30.784814 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 16 13:40:30.785037 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:30.784920 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 16 13:40:30.785037 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:30.785004 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wb4dz\"" Sep 16 13:40:30.799797 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:30.799776 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s"] Sep 16 13:40:30.929942 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:30.929913 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0f89b7-975b-400c-be50-27ccfd55302f-openshift-service-ca\") pod \"maas-api-f5f7c95d8-sq75s\" (UID: \"7f0f89b7-975b-400c-be50-27ccfd55302f\") " pod="odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s" Sep 16 13:40:30.930135 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:30.929986 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f0f89b7-975b-400c-be50-27ccfd55302f-metrics-certs\") pod \"maas-api-f5f7c95d8-sq75s\" (UID: \"7f0f89b7-975b-400c-be50-27ccfd55302f\") " pod="odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s" Sep 16 13:40:30.930135 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:30.930013 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkzx7\" (UniqueName: \"kubernetes.io/projected/7f0f89b7-975b-400c-be50-27ccfd55302f-kube-api-access-qkzx7\") pod \"maas-api-f5f7c95d8-sq75s\" (UID: \"7f0f89b7-975b-400c-be50-27ccfd55302f\") " pod="odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s" Sep 16 13:40:30.930135 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:30.930042 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f0f89b7-975b-400c-be50-27ccfd55302f-maas-api-tls\") pod \"maas-api-f5f7c95d8-sq75s\" (UID: \"7f0f89b7-975b-400c-be50-27ccfd55302f\") " pod="odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s" Sep 16 13:40:31.030514 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:31.030440 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f0f89b7-975b-400c-be50-27ccfd55302f-maas-api-tls\") pod \"maas-api-f5f7c95d8-sq75s\" (UID: \"7f0f89b7-975b-400c-be50-27ccfd55302f\") " pod="odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s" Sep 16 13:40:31.030514 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:31.030487 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0f89b7-975b-400c-be50-27ccfd55302f-openshift-service-ca\") pod \"maas-api-f5f7c95d8-sq75s\" (UID: \"7f0f89b7-975b-400c-be50-27ccfd55302f\") " pod="odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s" Sep 16 13:40:31.030724 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:31.030630 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f0f89b7-975b-400c-be50-27ccfd55302f-metrics-certs\") pod \"maas-api-f5f7c95d8-sq75s\" (UID: \"7f0f89b7-975b-400c-be50-27ccfd55302f\") " pod="odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s" Sep 16 13:40:31.030724 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:31.030688 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkzx7\" (UniqueName: \"kubernetes.io/projected/7f0f89b7-975b-400c-be50-27ccfd55302f-kube-api-access-qkzx7\") pod \"maas-api-f5f7c95d8-sq75s\" (UID: \"7f0f89b7-975b-400c-be50-27ccfd55302f\") " pod="odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s" Sep 16 13:40:31.031255 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:31.031235 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0f89b7-975b-400c-be50-27ccfd55302f-openshift-service-ca\") pod \"maas-api-f5f7c95d8-sq75s\" (UID: \"7f0f89b7-975b-400c-be50-27ccfd55302f\") " pod="odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s" Sep 16 13:40:31.033300 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:31.033260 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f0f89b7-975b-400c-be50-27ccfd55302f-metrics-certs\") pod \"maas-api-f5f7c95d8-sq75s\" (UID: \"7f0f89b7-975b-400c-be50-27ccfd55302f\") " pod="odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s" Sep 16 13:40:31.033779 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:31.033757 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f0f89b7-975b-400c-be50-27ccfd55302f-maas-api-tls\") pod \"maas-api-f5f7c95d8-sq75s\" (UID: \"7f0f89b7-975b-400c-be50-27ccfd55302f\") " pod="odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s" Sep 16 13:40:31.039687 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:31.039667 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkzx7\" (UniqueName: \"kubernetes.io/projected/7f0f89b7-975b-400c-be50-27ccfd55302f-kube-api-access-qkzx7\") pod \"maas-api-f5f7c95d8-sq75s\" (UID: \"7f0f89b7-975b-400c-be50-27ccfd55302f\") " pod="odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s" Sep 16 13:40:31.090133 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:31.090102 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s" Sep 16 13:40:31.389112 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:31.389091 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s"] Sep 16 13:40:31.391165 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:40:31.391140 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0f89b7_975b_400c_be50_27ccfd55302f.slice/crio-8df7c33afc39c0e785f3aaad322248319116c0ce72bf3a7e42ddeba0cdb83289 WatchSource:0}: Error finding container 8df7c33afc39c0e785f3aaad322248319116c0ce72bf3a7e42ddeba0cdb83289: Status 404 returned error can't find the container with id 8df7c33afc39c0e785f3aaad322248319116c0ce72bf3a7e42ddeba0cdb83289 Sep 16 13:40:32.354237 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:32.354157 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s" event={"ID":"7f0f89b7-975b-400c-be50-27ccfd55302f","Type":"ContainerStarted","Data":"8df7c33afc39c0e785f3aaad322248319116c0ce72bf3a7e42ddeba0cdb83289"} Sep 16 13:40:34.367591 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:34.367553 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s" event={"ID":"7f0f89b7-975b-400c-be50-27ccfd55302f","Type":"ContainerStarted","Data":"acf821133e4f6c055619c670a36df2a4115b5ff6dbf94b9c31d31281fafe08c3"} Sep 16 13:40:34.367948 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:34.367623 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s" Sep 16 13:40:34.391795 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:34.391745 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s" podStartSLOduration=1.582504631 podStartE2EDuration="4.39173098s" totalImagesPullingTime="2.809226349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:40:34.389066192 +0000 UTC m=+595.433369168" watchObservedRunningTime="2026-09-16 13:40:34.39173098 +0000 UTC m=+595.436033944" Sep 16 13:40:39.401169 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:39.401145 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4xmxv_c7828efb-6be7-4f7b-986f-645a2370163a/console-operator/1.log" Sep 16 13:40:39.402634 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:39.402611 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4xmxv_c7828efb-6be7-4f7b-986f-645a2370163a/console-operator/1.log" Sep 16 13:40:40.378703 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:40.378676 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s" Sep 16 13:40:41.412212 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:41.412175 2562 generic.go:356] "Generic (PLEG): container finished" podID="22fb052b-58e9-441f-8139-f34dcca5ffbb" containerID="b0c307d514eaa25e98dbb43d43b8980d79cd73685b56de854458f86230b47b48" exitCode=0 Sep 16 13:40:41.413353 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:41.413332 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" event={"ID":"22fb052b-58e9-441f-8139-f34dcca5ffbb","Type":"ContainerDied","Data":"b0c307d514eaa25e98dbb43d43b8980d79cd73685b56de854458f86230b47b48"} Sep 16 13:40:41.413685 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:41.413669 2562 scope.go:117] "RemoveContainer" containerID="b0c307d514eaa25e98dbb43d43b8980d79cd73685b56de854458f86230b47b48" Sep 16 13:40:41.414638 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:41.414615 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 16 13:40:41.414708 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:41.414692 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 16 13:40:41.414757 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:41.414741 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 16 13:40:41.925562 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:41.925468 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:40:42.420048 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:42.419998 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" event={"ID":"22fb052b-58e9-441f-8139-f34dcca5ffbb","Type":"ContainerStarted","Data":"91dd104086e9808df90fc90965839bef2c38a207ccc1be3dffdee874a97c5066"} Sep 16 13:40:42.778592 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:42.778510 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:40:42.779319 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:42.779295 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-tszg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 16 13:40:42.779366 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:42.779347 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" podUID="22fb052b-58e9-441f-8139-f34dcca5ffbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 16 13:40:43.779339 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:43.779305 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-tszg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 16 13:40:43.779734 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:43.779361 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" podUID="22fb052b-58e9-441f-8139-f34dcca5ffbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 16 13:40:44.779250 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:44.779216 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-tszg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 16 13:40:44.779442 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:44.779280 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" podUID="22fb052b-58e9-441f-8139-f34dcca5ffbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 16 13:40:45.778652 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:45.778622 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-tszg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 16 13:40:45.778833 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:45.778684 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" podUID="22fb052b-58e9-441f-8139-f34dcca5ffbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 16 13:40:46.778848 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:46.778813 2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-tszg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 16 13:40:46.779227 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:46.778866 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" podUID="22fb052b-58e9-441f-8139-f34dcca5ffbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 16 13:40:47.782635 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:47.782608 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:40:47.783020 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:47.782990 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:40:47.783592 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:47.783577 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tszg8" Sep 16 13:40:50.996964 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:50.996933 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58"] Sep 16 13:40:51.002527 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.002507 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" Sep 16 13:40:51.006855 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.006833 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 16 13:40:51.006963 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.006859 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 16 13:40:51.006963 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.006837 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-m5ll2\"" Sep 16 13:40:51.006963 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.006835 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 16 13:40:51.012750 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.012728 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58"] Sep 16 13:40:51.013131 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.013110 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a246ae83-c70b-4ca0-9056-39de31224074-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58\" (UID: \"a246ae83-c70b-4ca0-9056-39de31224074\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" Sep 16 13:40:51.013180 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.013145 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a246ae83-c70b-4ca0-9056-39de31224074-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58\" (UID: \"a246ae83-c70b-4ca0-9056-39de31224074\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" Sep 16 13:40:51.013261 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.013242 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a246ae83-c70b-4ca0-9056-39de31224074-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58\" (UID: \"a246ae83-c70b-4ca0-9056-39de31224074\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" Sep 16 13:40:51.013314 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.013299 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a246ae83-c70b-4ca0-9056-39de31224074-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58\" (UID: \"a246ae83-c70b-4ca0-9056-39de31224074\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" Sep 16 13:40:51.013371 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.013355 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm5cz\" (UniqueName: \"kubernetes.io/projected/a246ae83-c70b-4ca0-9056-39de31224074-kube-api-access-nm5cz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58\" (UID: \"a246ae83-c70b-4ca0-9056-39de31224074\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" Sep 16 13:40:51.013443 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.013429 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a246ae83-c70b-4ca0-9056-39de31224074-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58\" (UID: \"a246ae83-c70b-4ca0-9056-39de31224074\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" Sep 16 13:40:51.114847 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.114820 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm5cz\" (UniqueName: \"kubernetes.io/projected/a246ae83-c70b-4ca0-9056-39de31224074-kube-api-access-nm5cz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58\" (UID: \"a246ae83-c70b-4ca0-9056-39de31224074\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" Sep 16 13:40:51.115073 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.114860 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a246ae83-c70b-4ca0-9056-39de31224074-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58\" (UID: \"a246ae83-c70b-4ca0-9056-39de31224074\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" Sep 16 13:40:51.115073 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.114886 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a246ae83-c70b-4ca0-9056-39de31224074-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58\" (UID: \"a246ae83-c70b-4ca0-9056-39de31224074\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" Sep 16 13:40:51.115073 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.114903 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a246ae83-c70b-4ca0-9056-39de31224074-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58\" (UID: \"a246ae83-c70b-4ca0-9056-39de31224074\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" Sep 16 13:40:51.115073 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.114943 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a246ae83-c70b-4ca0-9056-39de31224074-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58\" (UID: \"a246ae83-c70b-4ca0-9056-39de31224074\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" Sep 16 13:40:51.115073 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.114970 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a246ae83-c70b-4ca0-9056-39de31224074-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58\" (UID: \"a246ae83-c70b-4ca0-9056-39de31224074\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" Sep 16 13:40:51.115351 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.115184 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a246ae83-c70b-4ca0-9056-39de31224074-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58\" (UID: \"a246ae83-c70b-4ca0-9056-39de31224074\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" Sep 16 13:40:51.115351 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.115265 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a246ae83-c70b-4ca0-9056-39de31224074-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58\" (UID: \"a246ae83-c70b-4ca0-9056-39de31224074\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" Sep 16 13:40:51.115351 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.115290 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a246ae83-c70b-4ca0-9056-39de31224074-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58\" (UID: \"a246ae83-c70b-4ca0-9056-39de31224074\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" Sep 16 13:40:51.117347 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.117330 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a246ae83-c70b-4ca0-9056-39de31224074-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58\" (UID: \"a246ae83-c70b-4ca0-9056-39de31224074\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" Sep 16 13:40:51.117518 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.117504 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a246ae83-c70b-4ca0-9056-39de31224074-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58\" (UID: \"a246ae83-c70b-4ca0-9056-39de31224074\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" Sep 16 13:40:51.124781 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.124756 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm5cz\" (UniqueName: \"kubernetes.io/projected/a246ae83-c70b-4ca0-9056-39de31224074-kube-api-access-nm5cz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58\" (UID: \"a246ae83-c70b-4ca0-9056-39de31224074\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" Sep 16 13:40:51.313828 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.313747 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" Sep 16 13:40:51.589139 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.589116 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58"] Sep 16 13:40:51.590786 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:40:51.590763 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda246ae83_c70b_4ca0_9056_39de31224074.slice/crio-bf9b45bdc9817e5caa977af268e2d1304dac270bb5a5e00d1d424f37a33b989c WatchSource:0}: Error finding container bf9b45bdc9817e5caa977af268e2d1304dac270bb5a5e00d1d424f37a33b989c: Status 404 returned error can't find the container with id bf9b45bdc9817e5caa977af268e2d1304dac270bb5a5e00d1d424f37a33b989c Sep 16 13:40:51.592637 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:51.592618 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 13:40:52.489772 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:52.489730 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" event={"ID":"a246ae83-c70b-4ca0-9056-39de31224074","Type":"ContainerStarted","Data":"bf9b45bdc9817e5caa977af268e2d1304dac270bb5a5e00d1d424f37a33b989c"} Sep 16 13:40:53.496291 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:53.496259 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" event={"ID":"a246ae83-c70b-4ca0-9056-39de31224074","Type":"ContainerStarted","Data":"4b5de83847303697b77e1ffdf757454ed44245c21e3668aac096b14440b80e18"} Sep 16 13:40:53.519036 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:40:53.518990 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" podStartSLOduration=2.468720758 podStartE2EDuration="3.518977645s" totalImagesPullingTime="1.050256887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:40:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:40:53.516783092 +0000 UTC m=+614.561086092" watchObservedRunningTime="2026-09-16 13:40:53.518977645 +0000 UTC m=+614.563280610" Sep 16 13:41:01.314914 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:41:01.314824 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" Sep 16 13:41:01.315444 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:41:01.314922 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" Sep 16 13:41:01.326862 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:41:01.326836 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" Sep 16 13:41:01.551043 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:41:01.551015 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58" Sep 16 13:43:15.754022 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:15.753984 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-679b96bc94-5swbb"] Sep 16 13:43:15.757750 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:15.757728 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-679b96bc94-5swbb" Sep 16 13:43:15.761737 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:15.761715 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 16 13:43:15.762928 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:15.762909 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xfzjj\"" Sep 16 13:43:15.774658 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:15.774640 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-679b96bc94-5swbb"] Sep 16 13:43:15.897475 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:15.897446 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bbb8d7ff-cded-411c-8b95-1e2eb0751441-tls-cert\") pod \"authorino-679b96bc94-5swbb\" (UID: \"bbb8d7ff-cded-411c-8b95-1e2eb0751441\") " pod="kuadrant-system/authorino-679b96bc94-5swbb" Sep 16 13:43:15.897622 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:15.897511 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrzch\" (UniqueName: \"kubernetes.io/projected/bbb8d7ff-cded-411c-8b95-1e2eb0751441-kube-api-access-zrzch\") pod \"authorino-679b96bc94-5swbb\" (UID: \"bbb8d7ff-cded-411c-8b95-1e2eb0751441\") " pod="kuadrant-system/authorino-679b96bc94-5swbb" Sep 16 13:43:15.998813 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:15.998781 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrzch\" (UniqueName: \"kubernetes.io/projected/bbb8d7ff-cded-411c-8b95-1e2eb0751441-kube-api-access-zrzch\") pod \"authorino-679b96bc94-5swbb\" (UID: \"bbb8d7ff-cded-411c-8b95-1e2eb0751441\") " pod="kuadrant-system/authorino-679b96bc94-5swbb" Sep 16 13:43:15.998959 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:15.998845 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bbb8d7ff-cded-411c-8b95-1e2eb0751441-tls-cert\") pod \"authorino-679b96bc94-5swbb\" (UID: \"bbb8d7ff-cded-411c-8b95-1e2eb0751441\") " pod="kuadrant-system/authorino-679b96bc94-5swbb" Sep 16 13:43:16.001432 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:16.001413 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bbb8d7ff-cded-411c-8b95-1e2eb0751441-tls-cert\") pod \"authorino-679b96bc94-5swbb\" (UID: \"bbb8d7ff-cded-411c-8b95-1e2eb0751441\") " pod="kuadrant-system/authorino-679b96bc94-5swbb" Sep 16 13:43:16.012384 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:16.012318 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrzch\" (UniqueName: \"kubernetes.io/projected/bbb8d7ff-cded-411c-8b95-1e2eb0751441-kube-api-access-zrzch\") pod \"authorino-679b96bc94-5swbb\" (UID: \"bbb8d7ff-cded-411c-8b95-1e2eb0751441\") " pod="kuadrant-system/authorino-679b96bc94-5swbb" Sep 16 13:43:16.066882 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:16.066860 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-679b96bc94-5swbb" Sep 16 13:43:16.338545 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:16.338523 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-679b96bc94-5swbb"] Sep 16 13:43:16.340660 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:43:16.340636 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbb8d7ff_cded_411c_8b95_1e2eb0751441.slice/crio-053ffd8296b4763ce808666fd577dadc3689744995a416130758b2f57bbfce32 WatchSource:0}: Error finding container 053ffd8296b4763ce808666fd577dadc3689744995a416130758b2f57bbfce32: Status 404 returned error can't find the container with id 053ffd8296b4763ce808666fd577dadc3689744995a416130758b2f57bbfce32 Sep 16 13:43:17.268620 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:17.268594 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-679b96bc94-5swbb" event={"ID":"bbb8d7ff-cded-411c-8b95-1e2eb0751441","Type":"ContainerStarted","Data":"d4cf7a78ab8b6551b3f5b7e5a3a9128ce4d08979b2591a3e9dc47c067760d08c"} Sep 16 13:43:17.268891 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:17.268638 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-679b96bc94-5swbb" event={"ID":"bbb8d7ff-cded-411c-8b95-1e2eb0751441","Type":"ContainerStarted","Data":"053ffd8296b4763ce808666fd577dadc3689744995a416130758b2f57bbfce32"} Sep 16 13:43:17.287760 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:17.287717 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-679b96bc94-5swbb" podStartSLOduration=1.439159724 podStartE2EDuration="2.287703718s" totalImagesPullingTime="848.543994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:17.28750931 +0000 UTC m=+758.331812278" watchObservedRunningTime="2026-09-16 13:43:17.287703718 +0000 UTC m=+758.332006684" Sep 16 13:43:47.267449 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.267414 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv"] Sep 16 13:43:47.271041 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.271023 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.275073 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.275046 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-aca190-openshift-default-dockercfg-n46r8\"" Sep 16 13:43:47.346481 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.346456 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv"] Sep 16 13:43:47.361893 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.361867 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istio-envoy\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.362022 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.361914 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istio-token\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.362095 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.362023 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-workload-socket\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.362095 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.362064 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-credential-socket\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.362095 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.362090 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istiod-ca-cert\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.362264 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.362159 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6grvc\" (UniqueName: \"kubernetes.io/projected/45e60603-fc1d-4b9d-a9ac-492b7be491e7-kube-api-access-6grvc\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.362264 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.362241 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-workload-certs\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.362391 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.362296 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istio-data\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.362391 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.362331 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istio-podinfo\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.463136 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.463110 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6grvc\" (UniqueName: \"kubernetes.io/projected/45e60603-fc1d-4b9d-a9ac-492b7be491e7-kube-api-access-6grvc\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.463303 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.463164 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-workload-certs\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.463303 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.463201 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istio-data\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.463303 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.463233 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istio-podinfo\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.463303 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.463264 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istio-envoy\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.463303 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.463297 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istio-token\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.463574 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.463352 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-workload-socket\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.463574 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.463405 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-credential-socket\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.463574 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.463427 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istiod-ca-cert\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.463727 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.463632 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-workload-certs\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.463727 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.463697 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-workload-socket\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.463833 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.463753 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-credential-socket\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.464043 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.464019 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istio-data\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.464160 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.464045 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istiod-ca-cert\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.465883 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.465859 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istio-envoy\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.466083 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.466065 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istio-podinfo\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.528261 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.528192 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6grvc\" (UniqueName: \"kubernetes.io/projected/45e60603-fc1d-4b9d-a9ac-492b7be491e7-kube-api-access-6grvc\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.544082 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.544059 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istio-token\") pod \"e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.580556 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.580532 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:47.814171 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.814087 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl"] Sep 16 13:43:47.818849 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.818832 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.852087 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.852061 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4bb29c-openshift-default-dockercfg-8k4xc\"" Sep 16 13:43:47.866728 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.866706 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istio-podinfo\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.866821 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.866733 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-workload-socket\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.866821 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.866751 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-workload-certs\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.866821 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.866804 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istio-envoy\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.866924 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.866835 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istiod-ca-cert\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.866924 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.866885 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-credential-socket\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.866924 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.866914 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istio-data\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.867027 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.866950 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwcm5\" (UniqueName: \"kubernetes.io/projected/6aefc8e6-56eb-46a4-b9a5-129a198815b7-kube-api-access-nwcm5\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.867027 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.866974 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istio-token\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.968226 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.968205 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-workload-certs\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.968340 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.968246 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istio-envoy\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.968432 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.968405 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istiod-ca-cert\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.968496 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.968463 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-credential-socket\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.968553 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.968513 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istio-data\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.968607 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.968552 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwcm5\" (UniqueName: \"kubernetes.io/projected/6aefc8e6-56eb-46a4-b9a5-129a198815b7-kube-api-access-nwcm5\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.968607 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.968587 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istio-token\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.968858 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.968624 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-workload-certs\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.968858 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.968688 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istio-podinfo\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.968858 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.968715 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-workload-socket\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.969004 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.968969 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istio-data\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.969140 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.969114 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-credential-socket\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.969305 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.969270 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-workload-socket\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.969698 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.969660 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istiod-ca-cert\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.971170 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.971145 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istio-envoy\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:47.971579 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:47.971561 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istio-podinfo\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:48.007878 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:48.007852 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl"] Sep 16 13:43:48.046206 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:48.046182 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istio-token\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:48.057971 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:43:48.057940 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e60603_fc1d_4b9d_a9ac_492b7be491e7.slice/crio-3d2065fc63715cd0d4226f51aca7ded4a4bdb439dcb39cba821cc2e40c034880 WatchSource:0}: Error finding container 3d2065fc63715cd0d4226f51aca7ded4a4bdb439dcb39cba821cc2e40c034880: Status 404 returned error can't find the container with id 3d2065fc63715cd0d4226f51aca7ded4a4bdb439dcb39cba821cc2e40c034880 Sep 16 13:43:48.060208 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:48.060175 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 16 13:43:48.060298 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:48.060262 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 16 13:43:48.060360 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:48.060307 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 16 13:43:48.100327 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:48.100300 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv"] Sep 16 13:43:48.113309 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:48.113289 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwcm5\" (UniqueName: \"kubernetes.io/projected/6aefc8e6-56eb-46a4-b9a5-129a198815b7-kube-api-access-nwcm5\") pod \"e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:48.128884 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:48.128866 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:48.441470 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:48.441428 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" event={"ID":"45e60603-fc1d-4b9d-a9ac-492b7be491e7","Type":"ContainerStarted","Data":"64c050d84c74b3ae0a12a88ea0b1071df67eef24d6d6966962d019a5f3f84381"} Sep 16 13:43:48.441880 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:48.441484 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" event={"ID":"45e60603-fc1d-4b9d-a9ac-492b7be491e7","Type":"ContainerStarted","Data":"3d2065fc63715cd0d4226f51aca7ded4a4bdb439dcb39cba821cc2e40c034880"} Sep 16 13:43:48.485236 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:48.485200 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 16 13:43:48.485356 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:48.485284 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 16 13:43:48.485356 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:48.485324 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 16 13:43:48.486825 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:48.486800 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl"] Sep 16 13:43:48.581179 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:48.581149 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:49.448874 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:49.448839 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" event={"ID":"6aefc8e6-56eb-46a4-b9a5-129a198815b7","Type":"ContainerStarted","Data":"423a591799bb20087eb6a2209703e5646d22f76d2c3f90099f881ddd0350eba5"} Sep 16 13:43:49.449276 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:49.448889 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" event={"ID":"6aefc8e6-56eb-46a4-b9a5-129a198815b7","Type":"ContainerStarted","Data":"dbac988688fbbdef31956fbd4d10e365f7efc4305ba1d1e79d96d5ff3acaa94b"} Sep 16 13:43:49.484139 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:49.484095 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" podStartSLOduration=2.48407912 podStartE2EDuration="2.48407912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:48.541107076 +0000 UTC m=+789.585410072" watchObservedRunningTime="2026-09-16 13:43:49.48407912 +0000 UTC m=+790.528382085" Sep 16 13:43:49.485990 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:49.485957 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" podStartSLOduration=2.485941383 podStartE2EDuration="2.485941383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:49.481655956 +0000 UTC m=+790.525958933" watchObservedRunningTime="2026-09-16 13:43:49.485941383 +0000 UTC m=+790.530244351" Sep 16 13:43:49.585760 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:49.585727 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:50.130108 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:50.130075 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:50.135120 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:50.135100 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:50.455083 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:50.455051 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:50.455587 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:50.455179 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:50.455955 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:50.455933 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:43:50.456074 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:50.456057 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:43:54.541517 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:54.541483 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp"] Sep 16 13:43:54.546998 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:54.546978 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp" Sep 16 13:43:54.559601 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:54.559582 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-aca190-serving-cert\"" Sep 16 13:43:54.633861 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:54.633834 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cc0bbd2-504a-47d9-bb6b-320cee929918-maas-api-tls\") pod \"maas-api-e2e-inf-a-aca190-776685b4cf-55slp\" (UID: \"0cc0bbd2-504a-47d9-bb6b-320cee929918\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp" Sep 16 13:43:54.633978 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:54.633865 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc0bbd2-504a-47d9-bb6b-320cee929918-openshift-service-ca\") pod \"maas-api-e2e-inf-a-aca190-776685b4cf-55slp\" (UID: \"0cc0bbd2-504a-47d9-bb6b-320cee929918\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp" Sep 16 13:43:54.633978 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:54.633910 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvt5g\" (UniqueName: \"kubernetes.io/projected/0cc0bbd2-504a-47d9-bb6b-320cee929918-kube-api-access-fvt5g\") pod \"maas-api-e2e-inf-a-aca190-776685b4cf-55slp\" (UID: \"0cc0bbd2-504a-47d9-bb6b-320cee929918\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp" Sep 16 13:43:54.633978 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:54.633951 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cc0bbd2-504a-47d9-bb6b-320cee929918-metrics-certs\") pod \"maas-api-e2e-inf-a-aca190-776685b4cf-55slp\" (UID: \"0cc0bbd2-504a-47d9-bb6b-320cee929918\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp" Sep 16 13:43:54.695567 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:54.695539 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp"] Sep 16 13:43:54.735352 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:54.735326 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cc0bbd2-504a-47d9-bb6b-320cee929918-maas-api-tls\") pod \"maas-api-e2e-inf-a-aca190-776685b4cf-55slp\" (UID: \"0cc0bbd2-504a-47d9-bb6b-320cee929918\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp" Sep 16 13:43:54.735495 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:54.735355 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc0bbd2-504a-47d9-bb6b-320cee929918-openshift-service-ca\") pod \"maas-api-e2e-inf-a-aca190-776685b4cf-55slp\" (UID: \"0cc0bbd2-504a-47d9-bb6b-320cee929918\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp" Sep 16 13:43:54.735495 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:54.735403 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvt5g\" (UniqueName: \"kubernetes.io/projected/0cc0bbd2-504a-47d9-bb6b-320cee929918-kube-api-access-fvt5g\") pod \"maas-api-e2e-inf-a-aca190-776685b4cf-55slp\" (UID: \"0cc0bbd2-504a-47d9-bb6b-320cee929918\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp" Sep 16 13:43:54.735495 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:54.735452 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cc0bbd2-504a-47d9-bb6b-320cee929918-metrics-certs\") pod \"maas-api-e2e-inf-a-aca190-776685b4cf-55slp\" (UID: \"0cc0bbd2-504a-47d9-bb6b-320cee929918\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp" Sep 16 13:43:54.735968 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:54.735940 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc0bbd2-504a-47d9-bb6b-320cee929918-openshift-service-ca\") pod \"maas-api-e2e-inf-a-aca190-776685b4cf-55slp\" (UID: \"0cc0bbd2-504a-47d9-bb6b-320cee929918\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp" Sep 16 13:43:54.737877 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:54.737850 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cc0bbd2-504a-47d9-bb6b-320cee929918-metrics-certs\") pod \"maas-api-e2e-inf-a-aca190-776685b4cf-55slp\" (UID: \"0cc0bbd2-504a-47d9-bb6b-320cee929918\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp" Sep 16 13:43:54.738038 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:54.738016 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cc0bbd2-504a-47d9-bb6b-320cee929918-maas-api-tls\") pod \"maas-api-e2e-inf-a-aca190-776685b4cf-55slp\" (UID: \"0cc0bbd2-504a-47d9-bb6b-320cee929918\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp" Sep 16 13:43:54.773785 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:54.773754 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvt5g\" (UniqueName: \"kubernetes.io/projected/0cc0bbd2-504a-47d9-bb6b-320cee929918-kube-api-access-fvt5g\") pod \"maas-api-e2e-inf-a-aca190-776685b4cf-55slp\" (UID: \"0cc0bbd2-504a-47d9-bb6b-320cee929918\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp" Sep 16 13:43:54.857929 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:54.857856 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp" Sep 16 13:43:55.151164 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:55.151132 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp"] Sep 16 13:43:55.152486 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:43:55.152457 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc0bbd2_504a_47d9_bb6b_320cee929918.slice/crio-7fb8245ffb1cbd03370ab7bcea5985e9e3c2de0f95b8ed496b71d065088b9626 WatchSource:0}: Error finding container 7fb8245ffb1cbd03370ab7bcea5985e9e3c2de0f95b8ed496b71d065088b9626: Status 404 returned error can't find the container with id 7fb8245ffb1cbd03370ab7bcea5985e9e3c2de0f95b8ed496b71d065088b9626 Sep 16 13:43:55.481704 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:55.481671 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp" event={"ID":"0cc0bbd2-504a-47d9-bb6b-320cee929918","Type":"ContainerStarted","Data":"7fb8245ffb1cbd03370ab7bcea5985e9e3c2de0f95b8ed496b71d065088b9626"} Sep 16 13:43:56.492115 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:56.492084 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp" event={"ID":"0cc0bbd2-504a-47d9-bb6b-320cee929918","Type":"ContainerStarted","Data":"33abb7597b15723238c1abdbe82674893e5d2961336c407fef300efa7abd845e"} Sep 16 13:43:56.492644 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:56.492207 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp" Sep 16 13:43:56.561740 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:56.561685 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp" podStartSLOduration=1.8815189399999999 podStartE2EDuration="2.561665004s" totalImagesPullingTime="680.146064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:56.559962741 +0000 UTC m=+797.604265734" watchObservedRunningTime="2026-09-16 13:43:56.561665004 +0000 UTC m=+797.605967971" Sep 16 13:43:57.193989 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:57.193950 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv"] Sep 16 13:43:57.197673 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:57.197652 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" Sep 16 13:43:57.200448 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:57.200425 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-46ccbd-serving-cert\"" Sep 16 13:43:57.235742 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:57.235717 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv"] Sep 16 13:43:57.261658 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:57.261635 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bbd6dfb-b359-433c-8cc2-3cc44869985b-maas-api-tls\") pod \"maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv\" (UID: \"4bbd6dfb-b359-433c-8cc2-3cc44869985b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" Sep 16 13:43:57.261784 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:57.261669 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbd6dfb-b359-433c-8cc2-3cc44869985b-metrics-certs\") pod \"maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv\" (UID: \"4bbd6dfb-b359-433c-8cc2-3cc44869985b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" Sep 16 13:43:57.261784 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:57.261729 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bbd6dfb-b359-433c-8cc2-3cc44869985b-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv\" (UID: \"4bbd6dfb-b359-433c-8cc2-3cc44869985b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" Sep 16 13:43:57.261884 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:57.261799 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr6hf\" (UniqueName: \"kubernetes.io/projected/4bbd6dfb-b359-433c-8cc2-3cc44869985b-kube-api-access-wr6hf\") pod \"maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv\" (UID: \"4bbd6dfb-b359-433c-8cc2-3cc44869985b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" Sep 16 13:43:57.363310 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:57.363280 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bbd6dfb-b359-433c-8cc2-3cc44869985b-maas-api-tls\") pod \"maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv\" (UID: \"4bbd6dfb-b359-433c-8cc2-3cc44869985b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" Sep 16 13:43:57.363310 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:57.363312 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbd6dfb-b359-433c-8cc2-3cc44869985b-metrics-certs\") pod \"maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv\" (UID: \"4bbd6dfb-b359-433c-8cc2-3cc44869985b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" Sep 16 13:43:57.363580 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:57.363340 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bbd6dfb-b359-433c-8cc2-3cc44869985b-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv\" (UID: \"4bbd6dfb-b359-433c-8cc2-3cc44869985b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" Sep 16 13:43:57.363580 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:57.363369 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr6hf\" (UniqueName: \"kubernetes.io/projected/4bbd6dfb-b359-433c-8cc2-3cc44869985b-kube-api-access-wr6hf\") pod \"maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv\" (UID: \"4bbd6dfb-b359-433c-8cc2-3cc44869985b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" Sep 16 13:43:57.363580 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:43:57.363445 2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-serving-cert: secret "maas-api-e2e-worker-w0-46ccbd-serving-cert" not found Sep 16 13:43:57.363580 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:43:57.363506 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bbd6dfb-b359-433c-8cc2-3cc44869985b-maas-api-tls podName:4bbd6dfb-b359-433c-8cc2-3cc44869985b nodeName:}" failed. No retries permitted until 2026-09-16 13:43:57.863491033 +0000 UTC m=+798.907793982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4bbd6dfb-b359-433c-8cc2-3cc44869985b-maas-api-tls") pod "maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" (UID: "4bbd6dfb-b359-433c-8cc2-3cc44869985b") : secret "maas-api-e2e-worker-w0-46ccbd-serving-cert" not found Sep 16 13:43:57.363975 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:57.363955 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bbd6dfb-b359-433c-8cc2-3cc44869985b-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv\" (UID: \"4bbd6dfb-b359-433c-8cc2-3cc44869985b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" Sep 16 13:43:57.365872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:57.365848 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbd6dfb-b359-433c-8cc2-3cc44869985b-metrics-certs\") pod \"maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv\" (UID: \"4bbd6dfb-b359-433c-8cc2-3cc44869985b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" Sep 16 13:43:57.384822 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:57.384801 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr6hf\" (UniqueName: \"kubernetes.io/projected/4bbd6dfb-b359-433c-8cc2-3cc44869985b-kube-api-access-wr6hf\") pod \"maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv\" (UID: \"4bbd6dfb-b359-433c-8cc2-3cc44869985b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" Sep 16 13:43:57.869339 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:57.869305 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bbd6dfb-b359-433c-8cc2-3cc44869985b-maas-api-tls\") pod \"maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv\" (UID: \"4bbd6dfb-b359-433c-8cc2-3cc44869985b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" Sep 16 13:43:57.869707 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:43:57.869472 2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-serving-cert: secret "maas-api-e2e-worker-w0-46ccbd-serving-cert" not found Sep 16 13:43:57.869707 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:43:57.869533 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bbd6dfb-b359-433c-8cc2-3cc44869985b-maas-api-tls podName:4bbd6dfb-b359-433c-8cc2-3cc44869985b nodeName:}" failed. No retries permitted until 2026-09-16 13:43:58.869516062 +0000 UTC m=+799.913819007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4bbd6dfb-b359-433c-8cc2-3cc44869985b-maas-api-tls") pod "maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" (UID: "4bbd6dfb-b359-433c-8cc2-3cc44869985b") : secret "maas-api-e2e-worker-w0-46ccbd-serving-cert" not found Sep 16 13:43:58.880329 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:58.880294 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bbd6dfb-b359-433c-8cc2-3cc44869985b-maas-api-tls\") pod \"maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv\" (UID: \"4bbd6dfb-b359-433c-8cc2-3cc44869985b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" Sep 16 13:43:58.882745 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:58.882727 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bbd6dfb-b359-433c-8cc2-3cc44869985b-maas-api-tls\") pod \"maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv\" (UID: \"4bbd6dfb-b359-433c-8cc2-3cc44869985b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" Sep 16 13:43:59.008010 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:59.007973 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" Sep 16 13:43:59.302596 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:59.302575 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv"] Sep 16 13:43:59.304284 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:43:59.304255 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bbd6dfb_b359_433c_8cc2_3cc44869985b.slice/crio-84590811f3aea733b47a8979192c9f08ea8d99548ec02e568b5f9dfa67c97195 WatchSource:0}: Error finding container 84590811f3aea733b47a8979192c9f08ea8d99548ec02e568b5f9dfa67c97195: Status 404 returned error can't find the container with id 84590811f3aea733b47a8979192c9f08ea8d99548ec02e568b5f9dfa67c97195 Sep 16 13:43:59.372043 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:59.372017 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s"] Sep 16 13:43:59.376605 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:59.376589 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" Sep 16 13:43:59.385342 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:59.385321 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-4bb29c-serving-cert\"" Sep 16 13:43:59.398416 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:59.398396 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s"] Sep 16 13:43:59.485409 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:59.485354 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37a19ec4-2357-4035-888e-505953c31e3e-metrics-certs\") pod \"maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s\" (UID: \"37a19ec4-2357-4035-888e-505953c31e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" Sep 16 13:43:59.485582 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:59.485421 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37a19ec4-2357-4035-888e-505953c31e3e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s\" (UID: \"37a19ec4-2357-4035-888e-505953c31e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" Sep 16 13:43:59.485582 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:59.485491 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt7jr\" (UniqueName: \"kubernetes.io/projected/37a19ec4-2357-4035-888e-505953c31e3e-kube-api-access-dt7jr\") pod \"maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s\" (UID: \"37a19ec4-2357-4035-888e-505953c31e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" Sep 16 13:43:59.485691 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:59.485669 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37a19ec4-2357-4035-888e-505953c31e3e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s\" (UID: \"37a19ec4-2357-4035-888e-505953c31e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" Sep 16 13:43:59.509885 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:59.509848 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" event={"ID":"4bbd6dfb-b359-433c-8cc2-3cc44869985b","Type":"ContainerStarted","Data":"84590811f3aea733b47a8979192c9f08ea8d99548ec02e568b5f9dfa67c97195"} Sep 16 13:43:59.586783 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:59.586716 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37a19ec4-2357-4035-888e-505953c31e3e-metrics-certs\") pod \"maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s\" (UID: \"37a19ec4-2357-4035-888e-505953c31e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" Sep 16 13:43:59.586783 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:59.586758 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37a19ec4-2357-4035-888e-505953c31e3e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s\" (UID: \"37a19ec4-2357-4035-888e-505953c31e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" Sep 16 13:43:59.586964 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:59.586791 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt7jr\" (UniqueName: \"kubernetes.io/projected/37a19ec4-2357-4035-888e-505953c31e3e-kube-api-access-dt7jr\") pod \"maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s\" (UID: \"37a19ec4-2357-4035-888e-505953c31e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" Sep 16 13:43:59.586964 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:59.586853 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37a19ec4-2357-4035-888e-505953c31e3e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s\" (UID: \"37a19ec4-2357-4035-888e-505953c31e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" Sep 16 13:43:59.586964 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:43:59.586879 2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-serving-cert: secret "maas-api-e2e-worker-w4-4bb29c-serving-cert" not found Sep 16 13:43:59.586964 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:43:59.586953 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37a19ec4-2357-4035-888e-505953c31e3e-maas-api-tls podName:37a19ec4-2357-4035-888e-505953c31e3e nodeName:}" failed. No retries permitted until 2026-09-16 13:44:00.086932592 +0000 UTC m=+801.131235536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/37a19ec4-2357-4035-888e-505953c31e3e-maas-api-tls") pod "maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" (UID: "37a19ec4-2357-4035-888e-505953c31e3e") : secret "maas-api-e2e-worker-w4-4bb29c-serving-cert" not found Sep 16 13:43:59.587505 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:59.587482 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37a19ec4-2357-4035-888e-505953c31e3e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s\" (UID: \"37a19ec4-2357-4035-888e-505953c31e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" Sep 16 13:43:59.589340 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:59.589322 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37a19ec4-2357-4035-888e-505953c31e3e-metrics-certs\") pod \"maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s\" (UID: \"37a19ec4-2357-4035-888e-505953c31e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" Sep 16 13:43:59.642672 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:43:59.642647 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt7jr\" (UniqueName: \"kubernetes.io/projected/37a19ec4-2357-4035-888e-505953c31e3e-kube-api-access-dt7jr\") pod \"maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s\" (UID: \"37a19ec4-2357-4035-888e-505953c31e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" Sep 16 13:44:00.091156 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:00.091126 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37a19ec4-2357-4035-888e-505953c31e3e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s\" (UID: \"37a19ec4-2357-4035-888e-505953c31e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" Sep 16 13:44:00.093642 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:00.093622 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37a19ec4-2357-4035-888e-505953c31e3e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s\" (UID: \"37a19ec4-2357-4035-888e-505953c31e3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" Sep 16 13:44:00.286977 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:00.286944 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" Sep 16 13:44:00.518363 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:00.518326 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" event={"ID":"4bbd6dfb-b359-433c-8cc2-3cc44869985b","Type":"ContainerStarted","Data":"dcf416d9844cde654a9c6a54c481df9848136c68931eca96516bf2608f4f1b54"} Sep 16 13:44:00.518566 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:00.518445 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" Sep 16 13:44:00.584372 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:00.584341 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s"] Sep 16 13:44:00.584966 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:00.584925 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" podStartSLOduration=3.221367742 podStartE2EDuration="3.584908402s" totalImagesPullingTime="363.54066ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:00.580077244 +0000 UTC m=+801.624380230" watchObservedRunningTime="2026-09-16 13:44:00.584908402 +0000 UTC m=+801.629211367" Sep 16 13:44:00.593628 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:44:00.593598 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a19ec4_2357_4035_888e_505953c31e3e.slice/crio-8712a29f6afbdf387c3ef6f6445f0aa656a034f25b9edbd7b2ec644a419e9641 WatchSource:0}: Error finding container 8712a29f6afbdf387c3ef6f6445f0aa656a034f25b9edbd7b2ec644a419e9641: Status 404 returned error can't find the container with id 8712a29f6afbdf387c3ef6f6445f0aa656a034f25b9edbd7b2ec644a419e9641 Sep 16 13:44:01.527268 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:01.527227 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" event={"ID":"37a19ec4-2357-4035-888e-505953c31e3e","Type":"ContainerStarted","Data":"eb4c633e67d7b48cae520587b9c5e2ae30ca18c729a2f90a718a2cdf36df123e"} Sep 16 13:44:01.527677 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:01.527294 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" event={"ID":"37a19ec4-2357-4035-888e-505953c31e3e","Type":"ContainerStarted","Data":"8712a29f6afbdf387c3ef6f6445f0aa656a034f25b9edbd7b2ec644a419e9641"} Sep 16 13:44:01.527677 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:01.527530 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" Sep 16 13:44:01.584775 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:01.584730 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" podStartSLOduration=2.188712344 podStartE2EDuration="2.584714036s" totalImagesPullingTime="396.001692ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:01.583564282 +0000 UTC m=+802.627867267" watchObservedRunningTime="2026-09-16 13:44:01.584714036 +0000 UTC m=+802.629017038" Sep 16 13:44:02.503203 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:02.503178 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp" Sep 16 13:44:06.548591 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:06.548523 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" Sep 16 13:44:07.542869 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:07.542841 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" Sep 16 13:44:09.959689 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:09.959649 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l"] Sep 16 13:44:09.963664 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:09.963642 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" Sep 16 13:44:09.968306 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:09.968285 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-14f2a6-serving-cert\"" Sep 16 13:44:09.991070 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:09.991042 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l"] Sep 16 13:44:10.100849 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:10.100816 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a475aa-f0c2-4a7a-9472-77744a18155b-metrics-certs\") pod \"maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l\" (UID: \"f6a475aa-f0c2-4a7a-9472-77744a18155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" Sep 16 13:44:10.101011 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:10.100878 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a475aa-f0c2-4a7a-9472-77744a18155b-openshift-service-ca\") pod \"maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l\" (UID: \"f6a475aa-f0c2-4a7a-9472-77744a18155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" Sep 16 13:44:10.101011 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:10.100945 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6a475aa-f0c2-4a7a-9472-77744a18155b-maas-api-tls\") pod \"maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l\" (UID: \"f6a475aa-f0c2-4a7a-9472-77744a18155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" Sep 16 13:44:10.101011 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:10.100984 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppbcl\" (UniqueName: \"kubernetes.io/projected/f6a475aa-f0c2-4a7a-9472-77744a18155b-kube-api-access-ppbcl\") pod \"maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l\" (UID: \"f6a475aa-f0c2-4a7a-9472-77744a18155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" Sep 16 13:44:10.202061 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:10.202017 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a475aa-f0c2-4a7a-9472-77744a18155b-metrics-certs\") pod \"maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l\" (UID: \"f6a475aa-f0c2-4a7a-9472-77744a18155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" Sep 16 13:44:10.202242 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:10.202098 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a475aa-f0c2-4a7a-9472-77744a18155b-openshift-service-ca\") pod \"maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l\" (UID: \"f6a475aa-f0c2-4a7a-9472-77744a18155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" Sep 16 13:44:10.202242 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:10.202184 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6a475aa-f0c2-4a7a-9472-77744a18155b-maas-api-tls\") pod \"maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l\" (UID: \"f6a475aa-f0c2-4a7a-9472-77744a18155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" Sep 16 13:44:10.202242 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:10.202235 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppbcl\" (UniqueName: \"kubernetes.io/projected/f6a475aa-f0c2-4a7a-9472-77744a18155b-kube-api-access-ppbcl\") pod \"maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l\" (UID: \"f6a475aa-f0c2-4a7a-9472-77744a18155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" Sep 16 13:44:10.202445 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:44:10.202333 2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-serving-cert: secret "maas-api-e2e-autores-14f2a6-serving-cert" not found Sep 16 13:44:10.202503 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:44:10.202446 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6a475aa-f0c2-4a7a-9472-77744a18155b-maas-api-tls podName:f6a475aa-f0c2-4a7a-9472-77744a18155b nodeName:}" failed. No retries permitted until 2026-09-16 13:44:10.702421104 +0000 UTC m=+811.746724071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f6a475aa-f0c2-4a7a-9472-77744a18155b-maas-api-tls") pod "maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" (UID: "f6a475aa-f0c2-4a7a-9472-77744a18155b") : secret "maas-api-e2e-autores-14f2a6-serving-cert" not found Sep 16 13:44:10.202901 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:10.202881 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a475aa-f0c2-4a7a-9472-77744a18155b-openshift-service-ca\") pod \"maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l\" (UID: \"f6a475aa-f0c2-4a7a-9472-77744a18155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" Sep 16 13:44:10.204704 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:10.204671 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a475aa-f0c2-4a7a-9472-77744a18155b-metrics-certs\") pod \"maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l\" (UID: \"f6a475aa-f0c2-4a7a-9472-77744a18155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" Sep 16 13:44:10.216298 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:10.216240 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppbcl\" (UniqueName: \"kubernetes.io/projected/f6a475aa-f0c2-4a7a-9472-77744a18155b-kube-api-access-ppbcl\") pod \"maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l\" (UID: \"f6a475aa-f0c2-4a7a-9472-77744a18155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" Sep 16 13:44:10.708258 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:10.708226 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6a475aa-f0c2-4a7a-9472-77744a18155b-maas-api-tls\") pod \"maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l\" (UID: \"f6a475aa-f0c2-4a7a-9472-77744a18155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" Sep 16 13:44:10.710788 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:10.710754 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6a475aa-f0c2-4a7a-9472-77744a18155b-maas-api-tls\") pod \"maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l\" (UID: \"f6a475aa-f0c2-4a7a-9472-77744a18155b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" Sep 16 13:44:10.875041 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:10.874961 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" Sep 16 13:44:11.184135 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:11.184103 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l"] Sep 16 13:44:11.199697 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:44:11.199668 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a475aa_f0c2_4a7a_9472_77744a18155b.slice/crio-74aa4fd0794b84495f90c07b207330de8f6d41633b4cde4fd82ae0d30cf97cab WatchSource:0}: Error finding container 74aa4fd0794b84495f90c07b207330de8f6d41633b4cde4fd82ae0d30cf97cab: Status 404 returned error can't find the container with id 74aa4fd0794b84495f90c07b207330de8f6d41633b4cde4fd82ae0d30cf97cab Sep 16 13:44:11.595063 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:11.594977 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" event={"ID":"f6a475aa-f0c2-4a7a-9472-77744a18155b","Type":"ContainerStarted","Data":"74aa4fd0794b84495f90c07b207330de8f6d41633b4cde4fd82ae0d30cf97cab"} Sep 16 13:44:11.596591 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:11.596569 2562 generic.go:356] "Generic (PLEG): container finished" podID="45e60603-fc1d-4b9d-a9ac-492b7be491e7" containerID="64c050d84c74b3ae0a12a88ea0b1071df67eef24d6d6966962d019a5f3f84381" exitCode=0 Sep 16 13:44:11.596714 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:11.596640 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" event={"ID":"45e60603-fc1d-4b9d-a9ac-492b7be491e7","Type":"ContainerDied","Data":"64c050d84c74b3ae0a12a88ea0b1071df67eef24d6d6966962d019a5f3f84381"} Sep 16 13:44:11.597040 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:11.597028 2562 scope.go:117] "RemoveContainer" containerID="64c050d84c74b3ae0a12a88ea0b1071df67eef24d6d6966962d019a5f3f84381" Sep 16 13:44:11.598110 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:11.598083 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 16 13:44:11.598206 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:11.598147 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 16 13:44:11.598206 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:11.598185 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 16 13:44:12.604024 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:12.603984 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" event={"ID":"f6a475aa-f0c2-4a7a-9472-77744a18155b","Type":"ContainerStarted","Data":"16e642a5f18a0cbb5ef21b9d81662966e46cb3721630129fe5f30d7d0bdd406c"} Sep 16 13:44:12.604524 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:12.604169 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" Sep 16 13:44:12.606145 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:12.606101 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" event={"ID":"45e60603-fc1d-4b9d-a9ac-492b7be491e7","Type":"ContainerStarted","Data":"be4ee03f11d3624c3db9f61489f7048013dfc5a2292995010238e24ec456144d"} Sep 16 13:44:12.670401 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:12.669953 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" podStartSLOduration=3.214250554 podStartE2EDuration="3.66992865s" totalImagesPullingTime="455.678096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:12.664815641 +0000 UTC m=+813.709118609" watchObservedRunningTime="2026-09-16 13:44:12.66992865 +0000 UTC m=+813.714231618" Sep 16 13:44:13.581008 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:13.580925 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:44:13.581311 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:13.581281 2562 patch_prober.go:28] interesting pod/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Sep 16 13:44:13.581560 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:13.581537 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" podUID="45e60603-fc1d-4b9d-a9ac-492b7be491e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Sep 16 13:44:14.581914 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:14.581861 2562 patch_prober.go:28] interesting pod/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Sep 16 13:44:14.582437 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:14.581998 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" podUID="45e60603-fc1d-4b9d-a9ac-492b7be491e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Sep 16 13:44:15.581115 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:15.581079 2562 patch_prober.go:28] interesting pod/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Sep 16 13:44:15.581297 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:15.581131 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" podUID="45e60603-fc1d-4b9d-a9ac-492b7be491e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Sep 16 13:44:16.581764 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:16.581687 2562 patch_prober.go:28] interesting pod/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Sep 16 13:44:16.581764 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:16.581744 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" podUID="45e60603-fc1d-4b9d-a9ac-492b7be491e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Sep 16 13:44:17.581760 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:17.581730 2562 patch_prober.go:28] interesting pod/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Sep 16 13:44:17.582161 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:17.581786 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" podUID="45e60603-fc1d-4b9d-a9ac-492b7be491e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Sep 16 13:44:17.602540 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:17.602505 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs"] Sep 16 13:44:17.606536 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:17.606514 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" Sep 16 13:44:17.609842 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:17.609823 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4e1907f7-serving-cert\"" Sep 16 13:44:17.643361 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:17.643329 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs"] Sep 16 13:44:17.689196 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:17.689158 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad49d62-9992-4fe9-875a-18cf749be936-openshift-service-ca\") pod \"maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs\" (UID: \"0ad49d62-9992-4fe9-875a-18cf749be936\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" Sep 16 13:44:17.689402 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:17.689292 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ad49d62-9992-4fe9-875a-18cf749be936-maas-api-tls\") pod \"maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs\" (UID: \"0ad49d62-9992-4fe9-875a-18cf749be936\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" Sep 16 13:44:17.689483 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:17.689368 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ad49d62-9992-4fe9-875a-18cf749be936-metrics-certs\") pod \"maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs\" (UID: \"0ad49d62-9992-4fe9-875a-18cf749be936\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" Sep 16 13:44:17.689483 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:17.689423 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq79l\" (UniqueName: \"kubernetes.io/projected/0ad49d62-9992-4fe9-875a-18cf749be936-kube-api-access-zq79l\") pod \"maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs\" (UID: \"0ad49d62-9992-4fe9-875a-18cf749be936\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" Sep 16 13:44:17.790536 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:17.790508 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ad49d62-9992-4fe9-875a-18cf749be936-maas-api-tls\") pod \"maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs\" (UID: \"0ad49d62-9992-4fe9-875a-18cf749be936\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" Sep 16 13:44:17.790719 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:17.790576 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ad49d62-9992-4fe9-875a-18cf749be936-metrics-certs\") pod \"maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs\" (UID: \"0ad49d62-9992-4fe9-875a-18cf749be936\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" Sep 16 13:44:17.790719 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:17.790602 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq79l\" (UniqueName: \"kubernetes.io/projected/0ad49d62-9992-4fe9-875a-18cf749be936-kube-api-access-zq79l\") pod \"maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs\" (UID: \"0ad49d62-9992-4fe9-875a-18cf749be936\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" Sep 16 13:44:17.790719 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:17.790648 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad49d62-9992-4fe9-875a-18cf749be936-openshift-service-ca\") pod \"maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs\" (UID: \"0ad49d62-9992-4fe9-875a-18cf749be936\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" Sep 16 13:44:17.790719 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:44:17.790692 2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-serving-cert: secret "maas-api-e2e-ait-4e1907f7-serving-cert" not found Sep 16 13:44:17.790932 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:44:17.790756 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ad49d62-9992-4fe9-875a-18cf749be936-maas-api-tls podName:0ad49d62-9992-4fe9-875a-18cf749be936 nodeName:}" failed. No retries permitted until 2026-09-16 13:44:18.290733561 +0000 UTC m=+819.335036507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ad49d62-9992-4fe9-875a-18cf749be936-maas-api-tls") pod "maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" (UID: "0ad49d62-9992-4fe9-875a-18cf749be936") : secret "maas-api-e2e-ait-4e1907f7-serving-cert" not found Sep 16 13:44:17.791405 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:17.791364 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad49d62-9992-4fe9-875a-18cf749be936-openshift-service-ca\") pod \"maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs\" (UID: \"0ad49d62-9992-4fe9-875a-18cf749be936\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" Sep 16 13:44:17.793510 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:17.793487 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ad49d62-9992-4fe9-875a-18cf749be936-metrics-certs\") pod \"maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs\" (UID: \"0ad49d62-9992-4fe9-875a-18cf749be936\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" Sep 16 13:44:17.807021 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:17.806995 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq79l\" (UniqueName: \"kubernetes.io/projected/0ad49d62-9992-4fe9-875a-18cf749be936-kube-api-access-zq79l\") pod \"maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs\" (UID: \"0ad49d62-9992-4fe9-875a-18cf749be936\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" Sep 16 13:44:18.296256 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:18.296212 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ad49d62-9992-4fe9-875a-18cf749be936-maas-api-tls\") pod \"maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs\" (UID: \"0ad49d62-9992-4fe9-875a-18cf749be936\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" Sep 16 13:44:18.296486 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:44:18.296369 2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-serving-cert: secret "maas-api-e2e-ait-4e1907f7-serving-cert" not found Sep 16 13:44:18.296486 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:44:18.296464 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ad49d62-9992-4fe9-875a-18cf749be936-maas-api-tls podName:0ad49d62-9992-4fe9-875a-18cf749be936 nodeName:}" failed. No retries permitted until 2026-09-16 13:44:19.29644883 +0000 UTC m=+820.340751774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ad49d62-9992-4fe9-875a-18cf749be936-maas-api-tls") pod "maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" (UID: "0ad49d62-9992-4fe9-875a-18cf749be936") : secret "maas-api-e2e-ait-4e1907f7-serving-cert" not found Sep 16 13:44:18.586048 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:18.585969 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:44:18.586556 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:18.586256 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:44:18.586898 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:18.586878 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:44:18.620186 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:18.620158 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" Sep 16 13:44:19.308161 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:19.308128 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ad49d62-9992-4fe9-875a-18cf749be936-maas-api-tls\") pod \"maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs\" (UID: \"0ad49d62-9992-4fe9-875a-18cf749be936\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" Sep 16 13:44:19.310669 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:19.310641 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ad49d62-9992-4fe9-875a-18cf749be936-maas-api-tls\") pod \"maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs\" (UID: \"0ad49d62-9992-4fe9-875a-18cf749be936\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" Sep 16 13:44:19.417427 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:19.417362 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" Sep 16 13:44:19.712561 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:19.712344 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs"] Sep 16 13:44:19.713836 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:44:19.713799 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad49d62_9992_4fe9_875a_18cf749be936.slice/crio-ce9069643f756169ffa9bab6677a5563013909586cf2b7b0f95e7613ac2c82c7 WatchSource:0}: Error finding container ce9069643f756169ffa9bab6677a5563013909586cf2b7b0f95e7613ac2c82c7: Status 404 returned error can't find the container with id ce9069643f756169ffa9bab6677a5563013909586cf2b7b0f95e7613ac2c82c7 Sep 16 13:44:20.461972 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:20.461930 2562 patch_prober.go:28] interesting pod/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 16 13:44:20.462169 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:20.461994 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" podUID="6aefc8e6-56eb-46a4-b9a5-129a198815b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 16 13:44:20.685597 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:20.685061 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" event={"ID":"0ad49d62-9992-4fe9-875a-18cf749be936","Type":"ContainerStarted","Data":"383879b159b22c0c6736dcad3f1f95901d14ec036d651b38059592aa32861f82"} Sep 16 13:44:20.685597 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:20.685141 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" event={"ID":"0ad49d62-9992-4fe9-875a-18cf749be936","Type":"ContainerStarted","Data":"ce9069643f756169ffa9bab6677a5563013909586cf2b7b0f95e7613ac2c82c7"} Sep 16 13:44:20.685597 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:20.685196 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" Sep 16 13:44:20.687397 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:20.687354 2562 generic.go:356] "Generic (PLEG): container finished" podID="6aefc8e6-56eb-46a4-b9a5-129a198815b7" containerID="423a591799bb20087eb6a2209703e5646d22f76d2c3f90099f881ddd0350eba5" exitCode=0 Sep 16 13:44:20.687528 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:20.687427 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" event={"ID":"6aefc8e6-56eb-46a4-b9a5-129a198815b7","Type":"ContainerDied","Data":"423a591799bb20087eb6a2209703e5646d22f76d2c3f90099f881ddd0350eba5"} Sep 16 13:44:20.687964 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:20.687947 2562 scope.go:117] "RemoveContainer" containerID="423a591799bb20087eb6a2209703e5646d22f76d2c3f90099f881ddd0350eba5" Sep 16 13:44:20.689408 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:20.689357 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 16 13:44:20.689481 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:20.689456 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 16 13:44:20.689520 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:20.689500 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 16 13:44:20.722218 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:20.722171 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" podStartSLOduration=3.306198731 podStartE2EDuration="3.722146889s" totalImagesPullingTime="415.948158ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:20.719286613 +0000 UTC m=+821.763589603" watchObservedRunningTime="2026-09-16 13:44:20.722146889 +0000 UTC m=+821.766449857" Sep 16 13:44:21.695628 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:21.695597 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" event={"ID":"6aefc8e6-56eb-46a4-b9a5-129a198815b7","Type":"ContainerStarted","Data":"dde57b1e1376160e38f4c9ac8607ce7ea6bc6a8d0e09e7c718ea5f06cb8ebd96"} Sep 16 13:44:22.129740 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:22.129654 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:44:22.130404 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:22.130300 2562 patch_prober.go:28] interesting pod/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 16 13:44:22.130593 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:22.130571 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" podUID="6aefc8e6-56eb-46a4-b9a5-129a198815b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 16 13:44:23.129844 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:23.129804 2562 patch_prober.go:28] interesting pod/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 16 13:44:23.130328 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:23.129869 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" podUID="6aefc8e6-56eb-46a4-b9a5-129a198815b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 16 13:44:23.279182 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:23.279139 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf"] Sep 16 13:44:23.281884 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:23.281863 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" Sep 16 13:44:23.284973 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:23.284955 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1e5db0-serving-cert\"" Sep 16 13:44:23.303219 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:23.303198 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf"] Sep 16 13:44:23.358025 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:23.357993 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37786cdb-af61-4416-8454-01255ca46b7f-maas-api-tls\") pod \"maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf\" (UID: \"37786cdb-af61-4416-8454-01255ca46b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" Sep 16 13:44:23.358025 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:23.358026 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37786cdb-af61-4416-8454-01255ca46b7f-metrics-certs\") pod \"maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf\" (UID: \"37786cdb-af61-4416-8454-01255ca46b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" Sep 16 13:44:23.358230 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:23.358132 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qdth\" (UniqueName: \"kubernetes.io/projected/37786cdb-af61-4416-8454-01255ca46b7f-kube-api-access-9qdth\") pod \"maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf\" (UID: \"37786cdb-af61-4416-8454-01255ca46b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" Sep 16 13:44:23.358230 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:23.358193 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37786cdb-af61-4416-8454-01255ca46b7f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf\" (UID: \"37786cdb-af61-4416-8454-01255ca46b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" Sep 16 13:44:23.459055 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:23.459019 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37786cdb-af61-4416-8454-01255ca46b7f-metrics-certs\") pod \"maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf\" (UID: \"37786cdb-af61-4416-8454-01255ca46b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" Sep 16 13:44:23.459289 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:23.459111 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qdth\" (UniqueName: \"kubernetes.io/projected/37786cdb-af61-4416-8454-01255ca46b7f-kube-api-access-9qdth\") pod \"maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf\" (UID: \"37786cdb-af61-4416-8454-01255ca46b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" Sep 16 13:44:23.459289 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:23.459158 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37786cdb-af61-4416-8454-01255ca46b7f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf\" (UID: \"37786cdb-af61-4416-8454-01255ca46b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" Sep 16 13:44:23.459289 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:23.459253 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37786cdb-af61-4416-8454-01255ca46b7f-maas-api-tls\") pod \"maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf\" (UID: \"37786cdb-af61-4416-8454-01255ca46b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" Sep 16 13:44:23.459508 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:44:23.459362 2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-serving-cert: secret "maas-api-e2e-inf-b-1e5db0-serving-cert" not found Sep 16 13:44:23.459508 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:44:23.459461 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37786cdb-af61-4416-8454-01255ca46b7f-maas-api-tls podName:37786cdb-af61-4416-8454-01255ca46b7f nodeName:}" failed. No retries permitted until 2026-09-16 13:44:23.959441224 +0000 UTC m=+825.003744182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/37786cdb-af61-4416-8454-01255ca46b7f-maas-api-tls") pod "maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" (UID: "37786cdb-af61-4416-8454-01255ca46b7f") : secret "maas-api-e2e-inf-b-1e5db0-serving-cert" not found Sep 16 13:44:23.459790 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:23.459768 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37786cdb-af61-4416-8454-01255ca46b7f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf\" (UID: \"37786cdb-af61-4416-8454-01255ca46b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" Sep 16 13:44:23.461527 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:23.461500 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37786cdb-af61-4416-8454-01255ca46b7f-metrics-certs\") pod \"maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf\" (UID: \"37786cdb-af61-4416-8454-01255ca46b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" Sep 16 13:44:23.484745 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:23.484719 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qdth\" (UniqueName: \"kubernetes.io/projected/37786cdb-af61-4416-8454-01255ca46b7f-kube-api-access-9qdth\") pod \"maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf\" (UID: \"37786cdb-af61-4416-8454-01255ca46b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" Sep 16 13:44:23.963939 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:23.963907 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37786cdb-af61-4416-8454-01255ca46b7f-maas-api-tls\") pod \"maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf\" (UID: \"37786cdb-af61-4416-8454-01255ca46b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" Sep 16 13:44:23.966442 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:23.966419 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37786cdb-af61-4416-8454-01255ca46b7f-maas-api-tls\") pod \"maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf\" (UID: \"37786cdb-af61-4416-8454-01255ca46b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" Sep 16 13:44:24.129792 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:24.129340 2562 patch_prober.go:28] interesting pod/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 16 13:44:24.129792 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:24.129429 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" podUID="6aefc8e6-56eb-46a4-b9a5-129a198815b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 16 13:44:24.192339 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:24.192313 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" Sep 16 13:44:24.516766 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:24.516740 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf"] Sep 16 13:44:24.517899 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:44:24.517878 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37786cdb_af61_4416_8454_01255ca46b7f.slice/crio-84992536b221604ef7033c9001707cdd6e205a69a49579a35091c472f9a947e6 WatchSource:0}: Error finding container 84992536b221604ef7033c9001707cdd6e205a69a49579a35091c472f9a947e6: Status 404 returned error can't find the container with id 84992536b221604ef7033c9001707cdd6e205a69a49579a35091c472f9a947e6 Sep 16 13:44:24.715104 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:24.715059 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" event={"ID":"37786cdb-af61-4416-8454-01255ca46b7f","Type":"ContainerStarted","Data":"84992536b221604ef7033c9001707cdd6e205a69a49579a35091c472f9a947e6"} Sep 16 13:44:25.133216 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:25.132736 2562 patch_prober.go:28] interesting pod/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 16 13:44:25.133216 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:25.132796 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" podUID="6aefc8e6-56eb-46a4-b9a5-129a198815b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 16 13:44:25.723642 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:25.723598 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" event={"ID":"37786cdb-af61-4416-8454-01255ca46b7f","Type":"ContainerStarted","Data":"3619870afe6ab30ec9cae228458f392d05af930a097ce46a4bbdba111201a400"} Sep 16 13:44:25.724490 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:25.724465 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" Sep 16 13:44:25.754588 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:25.754539 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" podStartSLOduration=2.452487277 podStartE2EDuration="2.754523887s" totalImagesPullingTime="302.03661ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:25.751483643 +0000 UTC m=+826.795786639" watchObservedRunningTime="2026-09-16 13:44:25.754523887 +0000 UTC m=+826.798826858" Sep 16 13:44:26.129887 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:26.129790 2562 patch_prober.go:28] interesting pod/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 16 13:44:26.129887 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:26.129859 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" podUID="6aefc8e6-56eb-46a4-b9a5-129a198815b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 16 13:44:26.701393 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:26.701360 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" Sep 16 13:44:27.134181 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:27.134096 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:44:27.134534 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:27.134514 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:44:27.135173 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:27.135155 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:44:32.741167 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:32.741137 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" Sep 16 13:44:51.000650 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:51.000618 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs"] Sep 16 13:44:51.001162 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:51.000852 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" podUID="0ad49d62-9992-4fe9-875a-18cf749be936" containerName="maas-api" containerID="cri-o://383879b159b22c0c6736dcad3f1f95901d14ec036d651b38059592aa32861f82" gracePeriod=30 Sep 16 13:44:51.251234 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:51.251176 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" Sep 16 13:44:51.321324 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:51.321298 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq79l\" (UniqueName: \"kubernetes.io/projected/0ad49d62-9992-4fe9-875a-18cf749be936-kube-api-access-zq79l\") pod \"0ad49d62-9992-4fe9-875a-18cf749be936\" (UID: \"0ad49d62-9992-4fe9-875a-18cf749be936\") " Sep 16 13:44:51.321532 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:51.321394 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ad49d62-9992-4fe9-875a-18cf749be936-metrics-certs\") pod \"0ad49d62-9992-4fe9-875a-18cf749be936\" (UID: \"0ad49d62-9992-4fe9-875a-18cf749be936\") " Sep 16 13:44:51.321532 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:51.321454 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad49d62-9992-4fe9-875a-18cf749be936-openshift-service-ca\") pod \"0ad49d62-9992-4fe9-875a-18cf749be936\" (UID: \"0ad49d62-9992-4fe9-875a-18cf749be936\") " Sep 16 13:44:51.321654 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:51.321561 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ad49d62-9992-4fe9-875a-18cf749be936-maas-api-tls\") pod \"0ad49d62-9992-4fe9-875a-18cf749be936\" (UID: \"0ad49d62-9992-4fe9-875a-18cf749be936\") " Sep 16 13:44:51.321937 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:51.321893 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad49d62-9992-4fe9-875a-18cf749be936-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0ad49d62-9992-4fe9-875a-18cf749be936" (UID: "0ad49d62-9992-4fe9-875a-18cf749be936"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:44:51.324426 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:51.324365 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ad49d62-9992-4fe9-875a-18cf749be936-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0ad49d62-9992-4fe9-875a-18cf749be936" (UID: "0ad49d62-9992-4fe9-875a-18cf749be936"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:44:51.324532 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:51.324421 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad49d62-9992-4fe9-875a-18cf749be936-kube-api-access-zq79l" (OuterVolumeSpecName: "kube-api-access-zq79l") pod "0ad49d62-9992-4fe9-875a-18cf749be936" (UID: "0ad49d62-9992-4fe9-875a-18cf749be936"). InnerVolumeSpecName "kube-api-access-zq79l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:44:51.324664 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:51.324641 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ad49d62-9992-4fe9-875a-18cf749be936-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0ad49d62-9992-4fe9-875a-18cf749be936" (UID: "0ad49d62-9992-4fe9-875a-18cf749be936"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:44:51.422528 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:51.422501 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ad49d62-9992-4fe9-875a-18cf749be936-metrics-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:44:51.422652 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:51.422531 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad49d62-9992-4fe9-875a-18cf749be936-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:44:51.422652 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:51.422546 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ad49d62-9992-4fe9-875a-18cf749be936-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:44:51.422652 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:51.422559 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq79l\" (UniqueName: \"kubernetes.io/projected/0ad49d62-9992-4fe9-875a-18cf749be936-kube-api-access-zq79l\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:44:51.880344 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:51.880290 2562 generic.go:356] "Generic (PLEG): container finished" podID="0ad49d62-9992-4fe9-875a-18cf749be936" containerID="383879b159b22c0c6736dcad3f1f95901d14ec036d651b38059592aa32861f82" exitCode=0 Sep 16 13:44:51.880684 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:51.880438 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" Sep 16 13:44:51.880684 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:51.880438 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" event={"ID":"0ad49d62-9992-4fe9-875a-18cf749be936","Type":"ContainerDied","Data":"383879b159b22c0c6736dcad3f1f95901d14ec036d651b38059592aa32861f82"} Sep 16 13:44:51.880684 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:51.880512 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs" event={"ID":"0ad49d62-9992-4fe9-875a-18cf749be936","Type":"ContainerDied","Data":"ce9069643f756169ffa9bab6677a5563013909586cf2b7b0f95e7613ac2c82c7"} Sep 16 13:44:51.880684 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:51.880543 2562 scope.go:117] "RemoveContainer" containerID="383879b159b22c0c6736dcad3f1f95901d14ec036d651b38059592aa32861f82" Sep 16 13:44:51.896506 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:51.896486 2562 scope.go:117] "RemoveContainer" containerID="383879b159b22c0c6736dcad3f1f95901d14ec036d651b38059592aa32861f82" Sep 16 13:44:51.896852 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:44:51.896829 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"383879b159b22c0c6736dcad3f1f95901d14ec036d651b38059592aa32861f82\": container with ID starting with 383879b159b22c0c6736dcad3f1f95901d14ec036d651b38059592aa32861f82 not found: ID does not exist" containerID="383879b159b22c0c6736dcad3f1f95901d14ec036d651b38059592aa32861f82" Sep 16 13:44:51.896959 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:51.896859 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"383879b159b22c0c6736dcad3f1f95901d14ec036d651b38059592aa32861f82"} err="failed to get container status \"383879b159b22c0c6736dcad3f1f95901d14ec036d651b38059592aa32861f82\": rpc error: code = NotFound desc = could not find container \"383879b159b22c0c6736dcad3f1f95901d14ec036d651b38059592aa32861f82\": container with ID starting with 383879b159b22c0c6736dcad3f1f95901d14ec036d651b38059592aa32861f82 not found: ID does not exist" Sep 16 13:44:51.903675 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:51.903653 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs"] Sep 16 13:44:51.909295 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:51.909263 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4e1907f7-68869bf496-q5gxs"] Sep 16 13:44:53.414181 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:53.414144 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ad49d62-9992-4fe9-875a-18cf749be936" path="/var/lib/kubelet/pods/0ad49d62-9992-4fe9-875a-18cf749be936/volumes" Sep 16 13:44:57.703443 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:57.703409 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr"] Sep 16 13:44:57.704051 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:57.704036 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ad49d62-9992-4fe9-875a-18cf749be936" containerName="maas-api" Sep 16 13:44:57.704094 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:57.704054 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ad49d62-9992-4fe9-875a-18cf749be936" containerName="maas-api" Sep 16 13:44:57.704191 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:57.704181 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ad49d62-9992-4fe9-875a-18cf749be936" containerName="maas-api" Sep 16 13:44:57.709262 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:57.709240 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" Sep 16 13:44:57.711858 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:57.711838 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-6d94e3f8-serving-cert\"" Sep 16 13:44:57.720773 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:57.720749 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr"] Sep 16 13:44:57.882715 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:57.882681 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1faa0e96-1003-44cc-8801-b5acc15edecb-openshift-service-ca\") pod \"maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr\" (UID: \"1faa0e96-1003-44cc-8801-b5acc15edecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" Sep 16 13:44:57.882916 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:57.882759 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxdr8\" (UniqueName: \"kubernetes.io/projected/1faa0e96-1003-44cc-8801-b5acc15edecb-kube-api-access-nxdr8\") pod \"maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr\" (UID: \"1faa0e96-1003-44cc-8801-b5acc15edecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" Sep 16 13:44:57.882916 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:57.882843 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1faa0e96-1003-44cc-8801-b5acc15edecb-maas-api-tls\") pod \"maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr\" (UID: \"1faa0e96-1003-44cc-8801-b5acc15edecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" Sep 16 13:44:57.883031 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:57.882916 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1faa0e96-1003-44cc-8801-b5acc15edecb-metrics-certs\") pod \"maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr\" (UID: \"1faa0e96-1003-44cc-8801-b5acc15edecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" Sep 16 13:44:57.984036 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:57.983967 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1faa0e96-1003-44cc-8801-b5acc15edecb-openshift-service-ca\") pod \"maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr\" (UID: \"1faa0e96-1003-44cc-8801-b5acc15edecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" Sep 16 13:44:57.984036 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:57.984026 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxdr8\" (UniqueName: \"kubernetes.io/projected/1faa0e96-1003-44cc-8801-b5acc15edecb-kube-api-access-nxdr8\") pod \"maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr\" (UID: \"1faa0e96-1003-44cc-8801-b5acc15edecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" Sep 16 13:44:57.984244 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:57.984055 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1faa0e96-1003-44cc-8801-b5acc15edecb-maas-api-tls\") pod \"maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr\" (UID: \"1faa0e96-1003-44cc-8801-b5acc15edecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" Sep 16 13:44:57.984244 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:57.984088 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1faa0e96-1003-44cc-8801-b5acc15edecb-metrics-certs\") pod \"maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr\" (UID: \"1faa0e96-1003-44cc-8801-b5acc15edecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" Sep 16 13:44:57.984244 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:44:57.984202 2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-serving-cert: secret "maas-api-e2e-migrate-6d94e3f8-serving-cert" not found Sep 16 13:44:57.984419 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:44:57.984269 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1faa0e96-1003-44cc-8801-b5acc15edecb-maas-api-tls podName:1faa0e96-1003-44cc-8801-b5acc15edecb nodeName:}" failed. No retries permitted until 2026-09-16 13:44:58.484249286 +0000 UTC m=+859.528552235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1faa0e96-1003-44cc-8801-b5acc15edecb-maas-api-tls") pod "maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" (UID: "1faa0e96-1003-44cc-8801-b5acc15edecb") : secret "maas-api-e2e-migrate-6d94e3f8-serving-cert" not found Sep 16 13:44:57.984608 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:57.984588 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1faa0e96-1003-44cc-8801-b5acc15edecb-openshift-service-ca\") pod \"maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr\" (UID: \"1faa0e96-1003-44cc-8801-b5acc15edecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" Sep 16 13:44:57.986673 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:57.986652 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1faa0e96-1003-44cc-8801-b5acc15edecb-metrics-certs\") pod \"maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr\" (UID: \"1faa0e96-1003-44cc-8801-b5acc15edecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" Sep 16 13:44:57.995825 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:57.995804 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxdr8\" (UniqueName: \"kubernetes.io/projected/1faa0e96-1003-44cc-8801-b5acc15edecb-kube-api-access-nxdr8\") pod \"maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr\" (UID: \"1faa0e96-1003-44cc-8801-b5acc15edecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" Sep 16 13:44:58.488809 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:58.488767 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1faa0e96-1003-44cc-8801-b5acc15edecb-maas-api-tls\") pod \"maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr\" (UID: \"1faa0e96-1003-44cc-8801-b5acc15edecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" Sep 16 13:44:58.491332 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:58.491305 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1faa0e96-1003-44cc-8801-b5acc15edecb-maas-api-tls\") pod \"maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr\" (UID: \"1faa0e96-1003-44cc-8801-b5acc15edecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" Sep 16 13:44:58.623224 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:58.623191 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" Sep 16 13:44:58.901195 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:58.901168 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr"] Sep 16 13:44:58.901861 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:44:58.901837 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1faa0e96_1003_44cc_8801_b5acc15edecb.slice/crio-0a4136ec8264d24b9b21674c4400018abc32722c1375ac0bde19bb1958ad07d9 WatchSource:0}: Error finding container 0a4136ec8264d24b9b21674c4400018abc32722c1375ac0bde19bb1958ad07d9: Status 404 returned error can't find the container with id 0a4136ec8264d24b9b21674c4400018abc32722c1375ac0bde19bb1958ad07d9 Sep 16 13:44:58.931303 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:58.931269 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" event={"ID":"1faa0e96-1003-44cc-8801-b5acc15edecb","Type":"ContainerStarted","Data":"0a4136ec8264d24b9b21674c4400018abc32722c1375ac0bde19bb1958ad07d9"} Sep 16 13:44:59.938460 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:59.938424 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" event={"ID":"1faa0e96-1003-44cc-8801-b5acc15edecb","Type":"ContainerStarted","Data":"b55ddb801f0099c0a6f0f23d7c54086a19dbdf907d14b45dc3f2dcd85d93221c"} Sep 16 13:44:59.938820 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:59.938567 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" Sep 16 13:44:59.960265 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:44:59.960228 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" podStartSLOduration=2.433298027 podStartE2EDuration="2.960214961s" totalImagesPullingTime="526.916934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:59.958203252 +0000 UTC m=+861.002506230" watchObservedRunningTime="2026-09-16 13:44:59.960214961 +0000 UTC m=+861.004517927" Sep 16 13:45:05.949962 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:05.949937 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" Sep 16 13:45:27.895301 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:27.895266 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr"] Sep 16 13:45:27.895761 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:27.895491 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" podUID="1faa0e96-1003-44cc-8801-b5acc15edecb" containerName="maas-api" containerID="cri-o://b55ddb801f0099c0a6f0f23d7c54086a19dbdf907d14b45dc3f2dcd85d93221c" gracePeriod=30 Sep 16 13:45:28.111594 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:28.111565 2562 generic.go:356] "Generic (PLEG): container finished" podID="1faa0e96-1003-44cc-8801-b5acc15edecb" containerID="b55ddb801f0099c0a6f0f23d7c54086a19dbdf907d14b45dc3f2dcd85d93221c" exitCode=0 Sep 16 13:45:28.111765 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:28.111638 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" event={"ID":"1faa0e96-1003-44cc-8801-b5acc15edecb","Type":"ContainerDied","Data":"b55ddb801f0099c0a6f0f23d7c54086a19dbdf907d14b45dc3f2dcd85d93221c"} Sep 16 13:45:28.150867 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:28.150812 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" Sep 16 13:45:28.267033 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:28.267000 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1faa0e96-1003-44cc-8801-b5acc15edecb-openshift-service-ca\") pod \"1faa0e96-1003-44cc-8801-b5acc15edecb\" (UID: \"1faa0e96-1003-44cc-8801-b5acc15edecb\") " Sep 16 13:45:28.267189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:28.267069 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1faa0e96-1003-44cc-8801-b5acc15edecb-maas-api-tls\") pod \"1faa0e96-1003-44cc-8801-b5acc15edecb\" (UID: \"1faa0e96-1003-44cc-8801-b5acc15edecb\") " Sep 16 13:45:28.267189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:28.267126 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxdr8\" (UniqueName: \"kubernetes.io/projected/1faa0e96-1003-44cc-8801-b5acc15edecb-kube-api-access-nxdr8\") pod \"1faa0e96-1003-44cc-8801-b5acc15edecb\" (UID: \"1faa0e96-1003-44cc-8801-b5acc15edecb\") " Sep 16 13:45:28.267189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:28.267145 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1faa0e96-1003-44cc-8801-b5acc15edecb-metrics-certs\") pod \"1faa0e96-1003-44cc-8801-b5acc15edecb\" (UID: \"1faa0e96-1003-44cc-8801-b5acc15edecb\") " Sep 16 13:45:28.267410 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:28.267339 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1faa0e96-1003-44cc-8801-b5acc15edecb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1faa0e96-1003-44cc-8801-b5acc15edecb" (UID: "1faa0e96-1003-44cc-8801-b5acc15edecb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:45:28.269418 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:28.269390 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1faa0e96-1003-44cc-8801-b5acc15edecb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1faa0e96-1003-44cc-8801-b5acc15edecb" (UID: "1faa0e96-1003-44cc-8801-b5acc15edecb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:45:28.269516 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:28.269418 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1faa0e96-1003-44cc-8801-b5acc15edecb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1faa0e96-1003-44cc-8801-b5acc15edecb" (UID: "1faa0e96-1003-44cc-8801-b5acc15edecb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:45:28.269516 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:28.269418 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1faa0e96-1003-44cc-8801-b5acc15edecb-kube-api-access-nxdr8" (OuterVolumeSpecName: "kube-api-access-nxdr8") pod "1faa0e96-1003-44cc-8801-b5acc15edecb" (UID: "1faa0e96-1003-44cc-8801-b5acc15edecb"). InnerVolumeSpecName "kube-api-access-nxdr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:45:28.367852 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:28.367819 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxdr8\" (UniqueName: \"kubernetes.io/projected/1faa0e96-1003-44cc-8801-b5acc15edecb-kube-api-access-nxdr8\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:45:28.367852 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:28.367847 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1faa0e96-1003-44cc-8801-b5acc15edecb-metrics-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:45:28.367852 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:28.367858 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1faa0e96-1003-44cc-8801-b5acc15edecb-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:45:28.368068 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:28.367867 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1faa0e96-1003-44cc-8801-b5acc15edecb-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:45:29.119070 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:29.119036 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" event={"ID":"1faa0e96-1003-44cc-8801-b5acc15edecb","Type":"ContainerDied","Data":"0a4136ec8264d24b9b21674c4400018abc32722c1375ac0bde19bb1958ad07d9"} Sep 16 13:45:29.119070 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:29.119068 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr" Sep 16 13:45:29.119533 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:29.119097 2562 scope.go:117] "RemoveContainer" containerID="b55ddb801f0099c0a6f0f23d7c54086a19dbdf907d14b45dc3f2dcd85d93221c" Sep 16 13:45:29.143556 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:29.143531 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr"] Sep 16 13:45:29.147279 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:29.147255 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6d94e3f8-65d66c6f-759fr"] Sep 16 13:45:29.416216 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:29.416188 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1faa0e96-1003-44cc-8801-b5acc15edecb" path="/var/lib/kubelet/pods/1faa0e96-1003-44cc-8801-b5acc15edecb/volumes" Sep 16 13:45:34.421910 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:34.421829 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv"] Sep 16 13:45:34.422536 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:34.422518 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1faa0e96-1003-44cc-8801-b5acc15edecb" containerName="maas-api" Sep 16 13:45:34.422594 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:34.422541 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1faa0e96-1003-44cc-8801-b5acc15edecb" containerName="maas-api" Sep 16 13:45:34.422692 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:34.422681 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1faa0e96-1003-44cc-8801-b5acc15edecb" containerName="maas-api" Sep 16 13:45:34.427642 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:34.427618 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" Sep 16 13:45:34.430467 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:34.430444 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1a60997c-serving-cert\"" Sep 16 13:45:34.438987 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:34.438963 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv"] Sep 16 13:45:34.525196 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:34.525163 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-metrics-certs\") pod \"maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv\" (UID: \"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" Sep 16 13:45:34.525366 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:34.525272 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-maas-api-tls\") pod \"maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv\" (UID: \"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" Sep 16 13:45:34.525457 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:34.525433 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mb6p\" (UniqueName: \"kubernetes.io/projected/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-kube-api-access-2mb6p\") pod \"maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv\" (UID: \"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" Sep 16 13:45:34.525511 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:34.525497 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-openshift-service-ca\") pod \"maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv\" (UID: \"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" Sep 16 13:45:34.626610 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:34.626577 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-metrics-certs\") pod \"maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv\" (UID: \"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" Sep 16 13:45:34.626780 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:34.626631 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-maas-api-tls\") pod \"maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv\" (UID: \"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" Sep 16 13:45:34.626780 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:34.626689 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mb6p\" (UniqueName: \"kubernetes.io/projected/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-kube-api-access-2mb6p\") pod \"maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv\" (UID: \"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" Sep 16 13:45:34.626780 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:34.626709 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-openshift-service-ca\") pod \"maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv\" (UID: \"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" Sep 16 13:45:34.626939 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:45:34.626791 2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-serving-cert: secret "maas-api-e2e-ait-1a60997c-serving-cert" not found Sep 16 13:45:34.626939 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:45:34.626868 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-maas-api-tls podName:0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b nodeName:}" failed. No retries permitted until 2026-09-16 13:45:35.126844395 +0000 UTC m=+896.171147364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-maas-api-tls") pod "maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" (UID: "0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b") : secret "maas-api-e2e-ait-1a60997c-serving-cert" not found Sep 16 13:45:34.627313 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:34.627292 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-openshift-service-ca\") pod \"maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv\" (UID: \"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" Sep 16 13:45:34.629121 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:34.629095 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-metrics-certs\") pod \"maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv\" (UID: \"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" Sep 16 13:45:34.653753 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:34.653729 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mb6p\" (UniqueName: \"kubernetes.io/projected/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-kube-api-access-2mb6p\") pod \"maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv\" (UID: \"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" Sep 16 13:45:35.132276 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:35.132247 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-maas-api-tls\") pod \"maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv\" (UID: \"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" Sep 16 13:45:35.134945 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:35.134915 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-maas-api-tls\") pod \"maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv\" (UID: \"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" Sep 16 13:45:35.343850 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:35.343809 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" Sep 16 13:45:35.656090 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:35.656048 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv"] Sep 16 13:45:35.663473 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:45:35.663442 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e1ee1fc_4e92_4cb8_ac63_2dd8efff701b.slice/crio-64c32f832bf2228365c84d276059c3616611e1feff8225901f80d9b840b6f565 WatchSource:0}: Error finding container 64c32f832bf2228365c84d276059c3616611e1feff8225901f80d9b840b6f565: Status 404 returned error can't find the container with id 64c32f832bf2228365c84d276059c3616611e1feff8225901f80d9b840b6f565 Sep 16 13:45:36.173436 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:36.173406 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" event={"ID":"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b","Type":"ContainerStarted","Data":"be51bee57a34e85d1d4f36a3ce754b27734c876b664df41a83153a29f9680091"} Sep 16 13:45:36.173620 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:36.173465 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" Sep 16 13:45:36.173620 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:36.173486 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" event={"ID":"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b","Type":"ContainerStarted","Data":"64c32f832bf2228365c84d276059c3616611e1feff8225901f80d9b840b6f565"} Sep 16 13:45:36.198152 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:36.198097 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" podStartSLOduration=1.821852413 podStartE2EDuration="2.198081603s" totalImagesPullingTime="376.22919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:36.194970332 +0000 UTC m=+897.239273317" watchObservedRunningTime="2026-09-16 13:45:36.198081603 +0000 UTC m=+897.242384568" Sep 16 13:45:39.444642 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:39.444605 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4xmxv_c7828efb-6be7-4f7b-986f-645a2370163a/console-operator/1.log" Sep 16 13:45:39.447327 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:39.447306 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4xmxv_c7828efb-6be7-4f7b-986f-645a2370163a/console-operator/1.log" Sep 16 13:45:42.184629 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:45:42.184602 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" Sep 16 13:46:13.446306 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:13.446277 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv"] Sep 16 13:46:13.446849 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:13.446495 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" podUID="0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b" containerName="maas-api" containerID="cri-o://be51bee57a34e85d1d4f36a3ce754b27734c876b664df41a83153a29f9680091" gracePeriod=30 Sep 16 13:46:13.689674 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:13.689649 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" Sep 16 13:46:13.811407 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:13.811299 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mb6p\" (UniqueName: \"kubernetes.io/projected/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-kube-api-access-2mb6p\") pod \"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b\" (UID: \"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b\") " Sep 16 13:46:13.811407 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:13.811365 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-maas-api-tls\") pod \"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b\" (UID: \"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b\") " Sep 16 13:46:13.811619 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:13.811471 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-metrics-certs\") pod \"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b\" (UID: \"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b\") " Sep 16 13:46:13.811619 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:13.811551 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-openshift-service-ca\") pod \"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b\" (UID: \"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b\") " Sep 16 13:46:13.811962 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:13.811914 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b" (UID: "0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:46:13.813672 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:13.813642 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b" (UID: "0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:46:13.813783 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:13.813695 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-kube-api-access-2mb6p" (OuterVolumeSpecName: "kube-api-access-2mb6p") pod "0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b" (UID: "0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b"). InnerVolumeSpecName "kube-api-access-2mb6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:46:13.813783 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:13.813705 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b" (UID: "0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:46:13.913751 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:13.913721 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:46:13.913751 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:13.913747 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-metrics-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:46:13.913751 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:13.913755 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:46:13.914018 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:13.913765 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mb6p\" (UniqueName: \"kubernetes.io/projected/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b-kube-api-access-2mb6p\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:46:14.415074 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:14.415045 2562 generic.go:356] "Generic (PLEG): container finished" podID="0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b" containerID="be51bee57a34e85d1d4f36a3ce754b27734c876b664df41a83153a29f9680091" exitCode=0 Sep 16 13:46:14.415276 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:14.415084 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" Sep 16 13:46:14.415276 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:14.415102 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" event={"ID":"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b","Type":"ContainerDied","Data":"be51bee57a34e85d1d4f36a3ce754b27734c876b664df41a83153a29f9680091"} Sep 16 13:46:14.415276 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:14.415149 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv" event={"ID":"0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b","Type":"ContainerDied","Data":"64c32f832bf2228365c84d276059c3616611e1feff8225901f80d9b840b6f565"} Sep 16 13:46:14.415276 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:14.415180 2562 scope.go:117] "RemoveContainer" containerID="be51bee57a34e85d1d4f36a3ce754b27734c876b664df41a83153a29f9680091" Sep 16 13:46:14.425486 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:14.425467 2562 scope.go:117] "RemoveContainer" containerID="be51bee57a34e85d1d4f36a3ce754b27734c876b664df41a83153a29f9680091" Sep 16 13:46:14.425807 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:46:14.425789 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be51bee57a34e85d1d4f36a3ce754b27734c876b664df41a83153a29f9680091\": container with ID starting with be51bee57a34e85d1d4f36a3ce754b27734c876b664df41a83153a29f9680091 not found: ID does not exist" containerID="be51bee57a34e85d1d4f36a3ce754b27734c876b664df41a83153a29f9680091" Sep 16 13:46:14.425859 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:14.425816 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be51bee57a34e85d1d4f36a3ce754b27734c876b664df41a83153a29f9680091"} err="failed to get container status \"be51bee57a34e85d1d4f36a3ce754b27734c876b664df41a83153a29f9680091\": rpc error: code = NotFound desc = could not find container \"be51bee57a34e85d1d4f36a3ce754b27734c876b664df41a83153a29f9680091\": container with ID starting with be51bee57a34e85d1d4f36a3ce754b27734c876b664df41a83153a29f9680091 not found: ID does not exist" Sep 16 13:46:14.439477 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:14.439459 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv"] Sep 16 13:46:14.444829 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:14.444808 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1a60997c-7c598867c7-6cxpv"] Sep 16 13:46:15.419856 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:15.419825 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b" path="/var/lib/kubelet/pods/0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b/volumes" Sep 16 13:46:24.065655 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:24.065622 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx"] Sep 16 13:46:24.066304 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:24.066282 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b" containerName="maas-api" Sep 16 13:46:24.066304 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:24.066306 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b" containerName="maas-api" Sep 16 13:46:24.066516 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:24.066486 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e1ee1fc-4e92-4cb8-ac63-2dd8efff701b" containerName="maas-api" Sep 16 13:46:24.071313 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:24.071293 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx" Sep 16 13:46:24.074215 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:24.074192 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-4985b031-serving-cert\"" Sep 16 13:46:24.081561 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:24.081542 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx"] Sep 16 13:46:24.109665 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:24.109641 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea363515-2502-4f8d-b9c1-fa119520901f-maas-api-tls\") pod \"maas-api-e2e-derive-4985b031-57bdd6b886-spmnx\" (UID: \"ea363515-2502-4f8d-b9c1-fa119520901f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx" Sep 16 13:46:24.109774 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:24.109670 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea363515-2502-4f8d-b9c1-fa119520901f-metrics-certs\") pod \"maas-api-e2e-derive-4985b031-57bdd6b886-spmnx\" (UID: \"ea363515-2502-4f8d-b9c1-fa119520901f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx" Sep 16 13:46:24.109774 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:24.109762 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bj2j\" (UniqueName: \"kubernetes.io/projected/ea363515-2502-4f8d-b9c1-fa119520901f-kube-api-access-5bj2j\") pod \"maas-api-e2e-derive-4985b031-57bdd6b886-spmnx\" (UID: \"ea363515-2502-4f8d-b9c1-fa119520901f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx" Sep 16 13:46:24.109849 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:24.109783 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea363515-2502-4f8d-b9c1-fa119520901f-openshift-service-ca\") pod \"maas-api-e2e-derive-4985b031-57bdd6b886-spmnx\" (UID: \"ea363515-2502-4f8d-b9c1-fa119520901f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx" Sep 16 13:46:24.210328 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:24.210301 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bj2j\" (UniqueName: \"kubernetes.io/projected/ea363515-2502-4f8d-b9c1-fa119520901f-kube-api-access-5bj2j\") pod \"maas-api-e2e-derive-4985b031-57bdd6b886-spmnx\" (UID: \"ea363515-2502-4f8d-b9c1-fa119520901f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx" Sep 16 13:46:24.210466 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:24.210337 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea363515-2502-4f8d-b9c1-fa119520901f-openshift-service-ca\") pod \"maas-api-e2e-derive-4985b031-57bdd6b886-spmnx\" (UID: \"ea363515-2502-4f8d-b9c1-fa119520901f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx" Sep 16 13:46:24.210466 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:24.210435 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea363515-2502-4f8d-b9c1-fa119520901f-maas-api-tls\") pod \"maas-api-e2e-derive-4985b031-57bdd6b886-spmnx\" (UID: \"ea363515-2502-4f8d-b9c1-fa119520901f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx" Sep 16 13:46:24.210466 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:24.210460 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea363515-2502-4f8d-b9c1-fa119520901f-metrics-certs\") pod \"maas-api-e2e-derive-4985b031-57bdd6b886-spmnx\" (UID: \"ea363515-2502-4f8d-b9c1-fa119520901f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx" Sep 16 13:46:24.210930 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:24.210911 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea363515-2502-4f8d-b9c1-fa119520901f-openshift-service-ca\") pod \"maas-api-e2e-derive-4985b031-57bdd6b886-spmnx\" (UID: \"ea363515-2502-4f8d-b9c1-fa119520901f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx" Sep 16 13:46:24.213035 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:24.213014 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea363515-2502-4f8d-b9c1-fa119520901f-metrics-certs\") pod \"maas-api-e2e-derive-4985b031-57bdd6b886-spmnx\" (UID: \"ea363515-2502-4f8d-b9c1-fa119520901f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx" Sep 16 13:46:24.213355 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:24.213333 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea363515-2502-4f8d-b9c1-fa119520901f-maas-api-tls\") pod \"maas-api-e2e-derive-4985b031-57bdd6b886-spmnx\" (UID: \"ea363515-2502-4f8d-b9c1-fa119520901f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx" Sep 16 13:46:24.226313 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:24.226290 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bj2j\" (UniqueName: \"kubernetes.io/projected/ea363515-2502-4f8d-b9c1-fa119520901f-kube-api-access-5bj2j\") pod \"maas-api-e2e-derive-4985b031-57bdd6b886-spmnx\" (UID: \"ea363515-2502-4f8d-b9c1-fa119520901f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx" Sep 16 13:46:24.385189 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:24.385111 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx" Sep 16 13:46:24.664358 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:24.664331 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx"] Sep 16 13:46:24.665893 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:46:24.665867 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea363515_2502_4f8d_b9c1_fa119520901f.slice/crio-58e5c7e99286239c798c44435e9b71f1462d29349ab39e4947028a6414439bd6 WatchSource:0}: Error finding container 58e5c7e99286239c798c44435e9b71f1462d29349ab39e4947028a6414439bd6: Status 404 returned error can't find the container with id 58e5c7e99286239c798c44435e9b71f1462d29349ab39e4947028a6414439bd6 Sep 16 13:46:24.667198 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:24.667180 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 13:46:25.483470 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:25.483435 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx" event={"ID":"ea363515-2502-4f8d-b9c1-fa119520901f","Type":"ContainerStarted","Data":"0cff859ee9a20445a45ab4304249dfafafdae58d99ae07b46babbba9b4ddfe2d"} Sep 16 13:46:25.483836 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:25.483488 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx" event={"ID":"ea363515-2502-4f8d-b9c1-fa119520901f","Type":"ContainerStarted","Data":"58e5c7e99286239c798c44435e9b71f1462d29349ab39e4947028a6414439bd6"} Sep 16 13:46:25.483836 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:25.483518 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx" Sep 16 13:46:25.507271 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:25.507214 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx" podStartSLOduration=0.902075223 podStartE2EDuration="1.507198655s" totalImagesPullingTime="605.123432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:46:25.504549093 +0000 UTC m=+946.548852077" watchObservedRunningTime="2026-09-16 13:46:25.507198655 +0000 UTC m=+946.551501669" Sep 16 13:46:31.495851 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:31.495821 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx" Sep 16 13:46:35.177001 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:35.176966 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l"] Sep 16 13:46:35.177431 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:35.177149 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" podUID="f6a475aa-f0c2-4a7a-9472-77744a18155b" containerName="maas-api" containerID="cri-o://16e642a5f18a0cbb5ef21b9d81662966e46cb3721630129fe5f30d7d0bdd406c" gracePeriod=30 Sep 16 13:46:35.432683 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:35.432633 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" Sep 16 13:46:35.513737 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:35.513710 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a475aa-f0c2-4a7a-9472-77744a18155b-metrics-certs\") pod \"f6a475aa-f0c2-4a7a-9472-77744a18155b\" (UID: \"f6a475aa-f0c2-4a7a-9472-77744a18155b\") " Sep 16 13:46:35.513934 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:35.513767 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppbcl\" (UniqueName: \"kubernetes.io/projected/f6a475aa-f0c2-4a7a-9472-77744a18155b-kube-api-access-ppbcl\") pod \"f6a475aa-f0c2-4a7a-9472-77744a18155b\" (UID: \"f6a475aa-f0c2-4a7a-9472-77744a18155b\") " Sep 16 13:46:35.513934 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:35.513798 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6a475aa-f0c2-4a7a-9472-77744a18155b-maas-api-tls\") pod \"f6a475aa-f0c2-4a7a-9472-77744a18155b\" (UID: \"f6a475aa-f0c2-4a7a-9472-77744a18155b\") " Sep 16 13:46:35.513934 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:35.513833 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a475aa-f0c2-4a7a-9472-77744a18155b-openshift-service-ca\") pod \"f6a475aa-f0c2-4a7a-9472-77744a18155b\" (UID: \"f6a475aa-f0c2-4a7a-9472-77744a18155b\") " Sep 16 13:46:35.514226 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:35.514197 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a475aa-f0c2-4a7a-9472-77744a18155b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f6a475aa-f0c2-4a7a-9472-77744a18155b" (UID: "f6a475aa-f0c2-4a7a-9472-77744a18155b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:46:35.516110 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:35.516085 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a475aa-f0c2-4a7a-9472-77744a18155b-kube-api-access-ppbcl" (OuterVolumeSpecName: "kube-api-access-ppbcl") pod "f6a475aa-f0c2-4a7a-9472-77744a18155b" (UID: "f6a475aa-f0c2-4a7a-9472-77744a18155b"). InnerVolumeSpecName "kube-api-access-ppbcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:46:35.516110 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:35.516100 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a475aa-f0c2-4a7a-9472-77744a18155b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f6a475aa-f0c2-4a7a-9472-77744a18155b" (UID: "f6a475aa-f0c2-4a7a-9472-77744a18155b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:46:35.516256 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:35.516174 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a475aa-f0c2-4a7a-9472-77744a18155b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f6a475aa-f0c2-4a7a-9472-77744a18155b" (UID: "f6a475aa-f0c2-4a7a-9472-77744a18155b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:46:35.545299 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:35.545270 2562 generic.go:356] "Generic (PLEG): container finished" podID="f6a475aa-f0c2-4a7a-9472-77744a18155b" containerID="16e642a5f18a0cbb5ef21b9d81662966e46cb3721630129fe5f30d7d0bdd406c" exitCode=0 Sep 16 13:46:35.545448 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:35.545317 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" Sep 16 13:46:35.545448 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:35.545356 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" event={"ID":"f6a475aa-f0c2-4a7a-9472-77744a18155b","Type":"ContainerDied","Data":"16e642a5f18a0cbb5ef21b9d81662966e46cb3721630129fe5f30d7d0bdd406c"} Sep 16 13:46:35.545554 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:35.545445 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l" event={"ID":"f6a475aa-f0c2-4a7a-9472-77744a18155b","Type":"ContainerDied","Data":"74aa4fd0794b84495f90c07b207330de8f6d41633b4cde4fd82ae0d30cf97cab"} Sep 16 13:46:35.545554 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:35.545464 2562 scope.go:117] "RemoveContainer" containerID="16e642a5f18a0cbb5ef21b9d81662966e46cb3721630129fe5f30d7d0bdd406c" Sep 16 13:46:35.556062 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:35.556044 2562 scope.go:117] "RemoveContainer" containerID="16e642a5f18a0cbb5ef21b9d81662966e46cb3721630129fe5f30d7d0bdd406c" Sep 16 13:46:35.556332 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:46:35.556314 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16e642a5f18a0cbb5ef21b9d81662966e46cb3721630129fe5f30d7d0bdd406c\": container with ID starting with 16e642a5f18a0cbb5ef21b9d81662966e46cb3721630129fe5f30d7d0bdd406c not found: ID does not exist" containerID="16e642a5f18a0cbb5ef21b9d81662966e46cb3721630129fe5f30d7d0bdd406c" Sep 16 13:46:35.556430 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:35.556341 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16e642a5f18a0cbb5ef21b9d81662966e46cb3721630129fe5f30d7d0bdd406c"} err="failed to get container status \"16e642a5f18a0cbb5ef21b9d81662966e46cb3721630129fe5f30d7d0bdd406c\": rpc error: code = NotFound desc = could not find container \"16e642a5f18a0cbb5ef21b9d81662966e46cb3721630129fe5f30d7d0bdd406c\": container with ID starting with 16e642a5f18a0cbb5ef21b9d81662966e46cb3721630129fe5f30d7d0bdd406c not found: ID does not exist" Sep 16 13:46:35.567685 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:35.567665 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l"] Sep 16 13:46:35.585285 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:35.585256 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-14f2a6-548bf9d74b-4sn8l"] Sep 16 13:46:35.615356 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:35.615330 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6a475aa-f0c2-4a7a-9472-77744a18155b-metrics-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:46:35.615356 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:35.615352 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppbcl\" (UniqueName: \"kubernetes.io/projected/f6a475aa-f0c2-4a7a-9472-77744a18155b-kube-api-access-ppbcl\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:46:35.615585 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:35.615362 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6a475aa-f0c2-4a7a-9472-77744a18155b-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:46:35.615585 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:35.615372 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6a475aa-f0c2-4a7a-9472-77744a18155b-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:46:37.414197 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:37.414158 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6a475aa-f0c2-4a7a-9472-77744a18155b" path="/var/lib/kubelet/pods/f6a475aa-f0c2-4a7a-9472-77744a18155b/volumes" Sep 16 13:46:50.898156 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:50.898123 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx"] Sep 16 13:46:50.898819 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:50.898449 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx" podUID="ea363515-2502-4f8d-b9c1-fa119520901f" containerName="maas-api" containerID="cri-o://0cff859ee9a20445a45ab4304249dfafafdae58d99ae07b46babbba9b4ddfe2d" gracePeriod=30 Sep 16 13:46:51.133043 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:51.133022 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx" Sep 16 13:46:51.264114 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:51.264086 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea363515-2502-4f8d-b9c1-fa119520901f-metrics-certs\") pod \"ea363515-2502-4f8d-b9c1-fa119520901f\" (UID: \"ea363515-2502-4f8d-b9c1-fa119520901f\") " Sep 16 13:46:51.264285 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:51.264148 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea363515-2502-4f8d-b9c1-fa119520901f-openshift-service-ca\") pod \"ea363515-2502-4f8d-b9c1-fa119520901f\" (UID: \"ea363515-2502-4f8d-b9c1-fa119520901f\") " Sep 16 13:46:51.264285 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:51.264179 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea363515-2502-4f8d-b9c1-fa119520901f-maas-api-tls\") pod \"ea363515-2502-4f8d-b9c1-fa119520901f\" (UID: \"ea363515-2502-4f8d-b9c1-fa119520901f\") " Sep 16 13:46:51.264285 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:51.264208 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bj2j\" (UniqueName: \"kubernetes.io/projected/ea363515-2502-4f8d-b9c1-fa119520901f-kube-api-access-5bj2j\") pod \"ea363515-2502-4f8d-b9c1-fa119520901f\" (UID: \"ea363515-2502-4f8d-b9c1-fa119520901f\") " Sep 16 13:46:51.264558 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:51.264529 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea363515-2502-4f8d-b9c1-fa119520901f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ea363515-2502-4f8d-b9c1-fa119520901f" (UID: "ea363515-2502-4f8d-b9c1-fa119520901f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:46:51.264701 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:51.264604 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea363515-2502-4f8d-b9c1-fa119520901f-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:46:51.266511 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:51.266483 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea363515-2502-4f8d-b9c1-fa119520901f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ea363515-2502-4f8d-b9c1-fa119520901f" (UID: "ea363515-2502-4f8d-b9c1-fa119520901f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:46:51.266611 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:51.266578 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea363515-2502-4f8d-b9c1-fa119520901f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ea363515-2502-4f8d-b9c1-fa119520901f" (UID: "ea363515-2502-4f8d-b9c1-fa119520901f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:46:51.266687 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:51.266662 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea363515-2502-4f8d-b9c1-fa119520901f-kube-api-access-5bj2j" (OuterVolumeSpecName: "kube-api-access-5bj2j") pod "ea363515-2502-4f8d-b9c1-fa119520901f" (UID: "ea363515-2502-4f8d-b9c1-fa119520901f"). InnerVolumeSpecName "kube-api-access-5bj2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:46:51.365902 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:51.365874 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea363515-2502-4f8d-b9c1-fa119520901f-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:46:51.365902 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:51.365902 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bj2j\" (UniqueName: \"kubernetes.io/projected/ea363515-2502-4f8d-b9c1-fa119520901f-kube-api-access-5bj2j\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:46:51.366104 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:51.365916 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea363515-2502-4f8d-b9c1-fa119520901f-metrics-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:46:51.647511 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:51.647429 2562 generic.go:356] "Generic (PLEG): container finished" podID="ea363515-2502-4f8d-b9c1-fa119520901f" containerID="0cff859ee9a20445a45ab4304249dfafafdae58d99ae07b46babbba9b4ddfe2d" exitCode=0 Sep 16 13:46:51.647511 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:51.647470 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx" Sep 16 13:46:51.647746 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:51.647521 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx" event={"ID":"ea363515-2502-4f8d-b9c1-fa119520901f","Type":"ContainerDied","Data":"0cff859ee9a20445a45ab4304249dfafafdae58d99ae07b46babbba9b4ddfe2d"} Sep 16 13:46:51.647746 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:51.647586 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx" event={"ID":"ea363515-2502-4f8d-b9c1-fa119520901f","Type":"ContainerDied","Data":"58e5c7e99286239c798c44435e9b71f1462d29349ab39e4947028a6414439bd6"} Sep 16 13:46:51.647746 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:51.647616 2562 scope.go:117] "RemoveContainer" containerID="0cff859ee9a20445a45ab4304249dfafafdae58d99ae07b46babbba9b4ddfe2d" Sep 16 13:46:51.656957 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:51.656940 2562 scope.go:117] "RemoveContainer" containerID="0cff859ee9a20445a45ab4304249dfafafdae58d99ae07b46babbba9b4ddfe2d" Sep 16 13:46:51.657262 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:46:51.657236 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cff859ee9a20445a45ab4304249dfafafdae58d99ae07b46babbba9b4ddfe2d\": container with ID starting with 0cff859ee9a20445a45ab4304249dfafafdae58d99ae07b46babbba9b4ddfe2d not found: ID does not exist" containerID="0cff859ee9a20445a45ab4304249dfafafdae58d99ae07b46babbba9b4ddfe2d" Sep 16 13:46:51.657369 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:51.657273 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cff859ee9a20445a45ab4304249dfafafdae58d99ae07b46babbba9b4ddfe2d"} err="failed to get container status \"0cff859ee9a20445a45ab4304249dfafafdae58d99ae07b46babbba9b4ddfe2d\": rpc error: code = NotFound desc = could not find container \"0cff859ee9a20445a45ab4304249dfafafdae58d99ae07b46babbba9b4ddfe2d\": container with ID starting with 0cff859ee9a20445a45ab4304249dfafafdae58d99ae07b46babbba9b4ddfe2d not found: ID does not exist" Sep 16 13:46:51.671413 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:51.671390 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx"] Sep 16 13:46:51.674401 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:51.674364 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4985b031-57bdd6b886-spmnx"] Sep 16 13:46:53.414141 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:46:53.414113 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea363515-2502-4f8d-b9c1-fa119520901f" path="/var/lib/kubelet/pods/ea363515-2502-4f8d-b9c1-fa119520901f/volumes" Sep 16 13:47:09.203902 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:09.203825 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv"] Sep 16 13:47:09.204350 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:09.204334 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea363515-2502-4f8d-b9c1-fa119520901f" containerName="maas-api" Sep 16 13:47:09.204421 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:09.204352 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea363515-2502-4f8d-b9c1-fa119520901f" containerName="maas-api" Sep 16 13:47:09.204421 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:09.204361 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6a475aa-f0c2-4a7a-9472-77744a18155b" containerName="maas-api" Sep 16 13:47:09.204421 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:09.204366 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6a475aa-f0c2-4a7a-9472-77744a18155b" containerName="maas-api" Sep 16 13:47:09.204518 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:09.204455 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6a475aa-f0c2-4a7a-9472-77744a18155b" containerName="maas-api" Sep 16 13:47:09.204518 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:09.204464 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea363515-2502-4f8d-b9c1-fa119520901f" containerName="maas-api" Sep 16 13:47:09.207762 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:09.207742 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" Sep 16 13:47:09.210676 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:09.210656 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-4678b2-serving-cert\"" Sep 16 13:47:09.221669 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:09.221641 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv"] Sep 16 13:47:09.236046 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:09.236023 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-metrics-certs\") pod \"maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv\" (UID: \"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" Sep 16 13:47:09.236168 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:09.236071 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-maas-api-tls\") pod \"maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv\" (UID: \"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" Sep 16 13:47:09.236254 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:09.236227 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt6c7\" (UniqueName: \"kubernetes.io/projected/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-kube-api-access-kt6c7\") pod \"maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv\" (UID: \"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" Sep 16 13:47:09.236291 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:09.236278 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-openshift-service-ca\") pod \"maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv\" (UID: \"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" Sep 16 13:47:09.337199 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:09.337162 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt6c7\" (UniqueName: \"kubernetes.io/projected/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-kube-api-access-kt6c7\") pod \"maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv\" (UID: \"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" Sep 16 13:47:09.337199 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:09.337203 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-openshift-service-ca\") pod \"maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv\" (UID: \"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" Sep 16 13:47:09.337483 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:09.337246 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-metrics-certs\") pod \"maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv\" (UID: \"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" Sep 16 13:47:09.337483 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:09.337287 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-maas-api-tls\") pod \"maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv\" (UID: \"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" Sep 16 13:47:09.337483 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:47:09.337441 2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-serving-cert: secret "maas-api-e2e-explicit-4678b2-serving-cert" not found Sep 16 13:47:09.337641 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:47:09.337502 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-maas-api-tls podName:eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b nodeName:}" failed. No retries permitted until 2026-09-16 13:47:09.837487463 +0000 UTC m=+990.881790407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-maas-api-tls") pod "maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" (UID: "eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b") : secret "maas-api-e2e-explicit-4678b2-serving-cert" not found Sep 16 13:47:09.337918 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:09.337899 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-openshift-service-ca\") pod \"maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv\" (UID: \"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" Sep 16 13:47:09.339870 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:09.339847 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-metrics-certs\") pod \"maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv\" (UID: \"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" Sep 16 13:47:09.345793 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:09.345762 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt6c7\" (UniqueName: \"kubernetes.io/projected/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-kube-api-access-kt6c7\") pod \"maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv\" (UID: \"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" Sep 16 13:47:09.842591 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:09.842557 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-maas-api-tls\") pod \"maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv\" (UID: \"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" Sep 16 13:47:09.845062 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:09.845039 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-maas-api-tls\") pod \"maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv\" (UID: \"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" Sep 16 13:47:10.119119 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:10.119034 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" Sep 16 13:47:10.394398 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:10.394362 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv"] Sep 16 13:47:10.760117 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:10.760094 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" event={"ID":"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b","Type":"ContainerStarted","Data":"f0ec7db880521c07ef6a417855cf0a190f7ad34fdfe4e1a9d51533ccbf393ea4"} Sep 16 13:47:11.769037 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:11.769001 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" event={"ID":"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b","Type":"ContainerStarted","Data":"da39c65da5c1835c902cfd541e4f27f422df8e90d151d5ce84da4a0176f029ce"} Sep 16 13:47:11.769472 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:11.769144 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" Sep 16 13:47:11.787698 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:11.787642 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" podStartSLOduration=2.429997418 podStartE2EDuration="2.787624226s" totalImagesPullingTime="357.626808ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:47:11.787153221 +0000 UTC m=+992.831456197" watchObservedRunningTime="2026-09-16 13:47:11.787624226 +0000 UTC m=+992.831927192" Sep 16 13:47:17.781424 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:47:17.781368 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" Sep 16 13:48:36.310300 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:48:36.310210 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl"] Sep 16 13:48:36.314602 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:48:36.314577 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" Sep 16 13:48:36.317691 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:48:36.317674 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ab3aa7c0-serving-cert\"" Sep 16 13:48:36.328049 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:48:36.328030 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl"] Sep 16 13:48:36.361683 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:48:36.361653 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411dca39-436d-4b3f-b5d0-ac1b46de461d-maas-api-tls\") pod \"maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl\" (UID: \"411dca39-436d-4b3f-b5d0-ac1b46de461d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" Sep 16 13:48:36.361794 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:48:36.361703 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tmwd\" (UniqueName: \"kubernetes.io/projected/411dca39-436d-4b3f-b5d0-ac1b46de461d-kube-api-access-8tmwd\") pod \"maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl\" (UID: \"411dca39-436d-4b3f-b5d0-ac1b46de461d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" Sep 16 13:48:36.361839 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:48:36.361819 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411dca39-436d-4b3f-b5d0-ac1b46de461d-openshift-service-ca\") pod \"maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl\" (UID: \"411dca39-436d-4b3f-b5d0-ac1b46de461d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" Sep 16 13:48:36.361885 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:48:36.361873 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/411dca39-436d-4b3f-b5d0-ac1b46de461d-metrics-certs\") pod \"maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl\" (UID: \"411dca39-436d-4b3f-b5d0-ac1b46de461d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" Sep 16 13:48:36.463088 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:48:36.463040 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/411dca39-436d-4b3f-b5d0-ac1b46de461d-metrics-certs\") pod \"maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl\" (UID: \"411dca39-436d-4b3f-b5d0-ac1b46de461d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" Sep 16 13:48:36.463266 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:48:36.463102 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411dca39-436d-4b3f-b5d0-ac1b46de461d-maas-api-tls\") pod \"maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl\" (UID: \"411dca39-436d-4b3f-b5d0-ac1b46de461d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" Sep 16 13:48:36.463266 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:48:36.463161 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tmwd\" (UniqueName: \"kubernetes.io/projected/411dca39-436d-4b3f-b5d0-ac1b46de461d-kube-api-access-8tmwd\") pod \"maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl\" (UID: \"411dca39-436d-4b3f-b5d0-ac1b46de461d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" Sep 16 13:48:36.463403 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:48:36.463264 2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-serving-cert: secret "maas-api-e2e-mt-ab3aa7c0-serving-cert" not found Sep 16 13:48:36.463403 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:48:36.463330 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/411dca39-436d-4b3f-b5d0-ac1b46de461d-maas-api-tls podName:411dca39-436d-4b3f-b5d0-ac1b46de461d nodeName:}" failed. No retries permitted until 2026-09-16 13:48:36.963308133 +0000 UTC m=+1078.007611079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/411dca39-436d-4b3f-b5d0-ac1b46de461d-maas-api-tls") pod "maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" (UID: "411dca39-436d-4b3f-b5d0-ac1b46de461d") : secret "maas-api-e2e-mt-ab3aa7c0-serving-cert" not found Sep 16 13:48:36.463403 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:48:36.463267 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411dca39-436d-4b3f-b5d0-ac1b46de461d-openshift-service-ca\") pod \"maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl\" (UID: \"411dca39-436d-4b3f-b5d0-ac1b46de461d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" Sep 16 13:48:36.463869 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:48:36.463852 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411dca39-436d-4b3f-b5d0-ac1b46de461d-openshift-service-ca\") pod \"maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl\" (UID: \"411dca39-436d-4b3f-b5d0-ac1b46de461d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" Sep 16 13:48:36.465653 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:48:36.465636 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/411dca39-436d-4b3f-b5d0-ac1b46de461d-metrics-certs\") pod \"maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl\" (UID: \"411dca39-436d-4b3f-b5d0-ac1b46de461d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" Sep 16 13:48:36.475664 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:48:36.475638 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tmwd\" (UniqueName: \"kubernetes.io/projected/411dca39-436d-4b3f-b5d0-ac1b46de461d-kube-api-access-8tmwd\") pod \"maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl\" (UID: \"411dca39-436d-4b3f-b5d0-ac1b46de461d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" Sep 16 13:48:36.968118 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:48:36.968087 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411dca39-436d-4b3f-b5d0-ac1b46de461d-maas-api-tls\") pod \"maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl\" (UID: \"411dca39-436d-4b3f-b5d0-ac1b46de461d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" Sep 16 13:48:36.970486 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:48:36.970462 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411dca39-436d-4b3f-b5d0-ac1b46de461d-maas-api-tls\") pod \"maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl\" (UID: \"411dca39-436d-4b3f-b5d0-ac1b46de461d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" Sep 16 13:48:37.227040 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:48:37.226960 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" Sep 16 13:48:37.508120 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:48:37.508096 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl"] Sep 16 13:48:37.509469 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:48:37.509444 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod411dca39_436d_4b3f_b5d0_ac1b46de461d.slice/crio-4dafa6e309868c13480113d95cf923f40341865b933aa11752a9f18fe30f9714 WatchSource:0}: Error finding container 4dafa6e309868c13480113d95cf923f40341865b933aa11752a9f18fe30f9714: Status 404 returned error can't find the container with id 4dafa6e309868c13480113d95cf923f40341865b933aa11752a9f18fe30f9714 Sep 16 13:48:38.294740 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:48:38.294649 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" event={"ID":"411dca39-436d-4b3f-b5d0-ac1b46de461d","Type":"ContainerStarted","Data":"5b078daabf91103131a2049ea69651f78aa54874da968adcdc605bd3b23ea842"} Sep 16 13:48:38.294740 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:48:38.294700 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" event={"ID":"411dca39-436d-4b3f-b5d0-ac1b46de461d","Type":"ContainerStarted","Data":"4dafa6e309868c13480113d95cf923f40341865b933aa11752a9f18fe30f9714"} Sep 16 13:48:38.294934 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:48:38.294790 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" Sep 16 13:48:38.324874 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:48:38.324815 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" podStartSLOduration=1.830858587 podStartE2EDuration="2.324801838s" totalImagesPullingTime="493.943251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:48:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:48:38.31580666 +0000 UTC m=+1079.360109645" watchObservedRunningTime="2026-09-16 13:48:38.324801838 +0000 UTC m=+1079.369104803" Sep 16 13:48:44.306820 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:48:44.306793 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" Sep 16 13:49:09.181562 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.181532 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9"] Sep 16 13:49:09.185058 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.185038 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:49:09.189005 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.188984 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-46ccbd\"/\"kube-root-ca.crt\"" Sep 16 13:49:09.189225 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.189038 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-46ccbd\"/\"openshift-service-ca.crt\"" Sep 16 13:49:09.189352 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.189051 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-46ccbd\"/\"distinct-w0-46ccbd-kserve-self-signed-certs\"" Sep 16 13:49:09.189454 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.189071 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-46ccbd\"/\"default-dockercfg-crjnl\"" Sep 16 13:49:09.192894 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.192571 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9"] Sep 16 13:49:09.274267 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.274237 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13b282d7-4177-478e-b1de-49b9eb1da87b-home\") pod \"distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9\" (UID: \"13b282d7-4177-478e-b1de-49b9eb1da87b\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:49:09.274450 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.274278 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13b282d7-4177-478e-b1de-49b9eb1da87b-model-cache\") pod \"distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9\" (UID: \"13b282d7-4177-478e-b1de-49b9eb1da87b\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:49:09.274450 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.274338 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdr27\" (UniqueName: \"kubernetes.io/projected/13b282d7-4177-478e-b1de-49b9eb1da87b-kube-api-access-bdr27\") pod \"distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9\" (UID: \"13b282d7-4177-478e-b1de-49b9eb1da87b\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:49:09.274450 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.274399 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13b282d7-4177-478e-b1de-49b9eb1da87b-tmp-dir\") pod \"distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9\" (UID: \"13b282d7-4177-478e-b1de-49b9eb1da87b\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:49:09.274607 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.274451 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13b282d7-4177-478e-b1de-49b9eb1da87b-dshm\") pod \"distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9\" (UID: \"13b282d7-4177-478e-b1de-49b9eb1da87b\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:49:09.274607 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.274491 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13b282d7-4177-478e-b1de-49b9eb1da87b-tls-certs\") pod \"distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9\" (UID: \"13b282d7-4177-478e-b1de-49b9eb1da87b\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:49:09.374998 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.374959 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13b282d7-4177-478e-b1de-49b9eb1da87b-home\") pod \"distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9\" (UID: \"13b282d7-4177-478e-b1de-49b9eb1da87b\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:49:09.375195 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.375010 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13b282d7-4177-478e-b1de-49b9eb1da87b-model-cache\") pod \"distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9\" (UID: \"13b282d7-4177-478e-b1de-49b9eb1da87b\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:49:09.375195 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.375045 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdr27\" (UniqueName: \"kubernetes.io/projected/13b282d7-4177-478e-b1de-49b9eb1da87b-kube-api-access-bdr27\") pod \"distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9\" (UID: \"13b282d7-4177-478e-b1de-49b9eb1da87b\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:49:09.375195 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.375066 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13b282d7-4177-478e-b1de-49b9eb1da87b-tmp-dir\") pod \"distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9\" (UID: \"13b282d7-4177-478e-b1de-49b9eb1da87b\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:49:09.375195 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.375103 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13b282d7-4177-478e-b1de-49b9eb1da87b-dshm\") pod \"distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9\" (UID: \"13b282d7-4177-478e-b1de-49b9eb1da87b\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:49:09.375195 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.375128 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13b282d7-4177-478e-b1de-49b9eb1da87b-tls-certs\") pod \"distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9\" (UID: \"13b282d7-4177-478e-b1de-49b9eb1da87b\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:49:09.375503 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.375442 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13b282d7-4177-478e-b1de-49b9eb1da87b-home\") pod \"distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9\" (UID: \"13b282d7-4177-478e-b1de-49b9eb1da87b\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:49:09.375503 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.375466 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13b282d7-4177-478e-b1de-49b9eb1da87b-model-cache\") pod \"distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9\" (UID: \"13b282d7-4177-478e-b1de-49b9eb1da87b\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:49:09.375578 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.375508 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13b282d7-4177-478e-b1de-49b9eb1da87b-tmp-dir\") pod \"distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9\" (UID: \"13b282d7-4177-478e-b1de-49b9eb1da87b\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:49:09.377628 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.377603 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13b282d7-4177-478e-b1de-49b9eb1da87b-dshm\") pod \"distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9\" (UID: \"13b282d7-4177-478e-b1de-49b9eb1da87b\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:49:09.377744 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.377728 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13b282d7-4177-478e-b1de-49b9eb1da87b-tls-certs\") pod \"distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9\" (UID: \"13b282d7-4177-478e-b1de-49b9eb1da87b\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:49:09.382993 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.382967 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdr27\" (UniqueName: \"kubernetes.io/projected/13b282d7-4177-478e-b1de-49b9eb1da87b-kube-api-access-bdr27\") pod \"distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9\" (UID: \"13b282d7-4177-478e-b1de-49b9eb1da87b\") " pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:49:09.495146 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.495070 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:49:09.770007 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:09.769980 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9"] Sep 16 13:49:09.770973 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:49:09.770937 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13b282d7_4177_478e_b1de_49b9eb1da87b.slice/crio-229c2b0c812322e04ed6a59f26710731343a5511fef5e44c22856af1f2e7c564 WatchSource:0}: Error finding container 229c2b0c812322e04ed6a59f26710731343a5511fef5e44c22856af1f2e7c564: Status 404 returned error can't find the container with id 229c2b0c812322e04ed6a59f26710731343a5511fef5e44c22856af1f2e7c564 Sep 16 13:49:10.494022 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:10.493984 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" event={"ID":"13b282d7-4177-478e-b1de-49b9eb1da87b","Type":"ContainerStarted","Data":"b31aebd9e1d927b484c79073aaa718c7300ed4730ff7a289e0765e57a578f96c"} Sep 16 13:49:10.494022 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:10.494024 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" event={"ID":"13b282d7-4177-478e-b1de-49b9eb1da87b","Type":"ContainerStarted","Data":"229c2b0c812322e04ed6a59f26710731343a5511fef5e44c22856af1f2e7c564"} Sep 16 13:49:10.512986 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:10.512933 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" podStartSLOduration=1.512914267 podStartE2EDuration="1.512914267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:49:10.512353278 +0000 UTC m=+1111.556656273" watchObservedRunningTime="2026-09-16 13:49:10.512914267 +0000 UTC m=+1111.557217233" Sep 16 13:49:19.496188 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:19.496155 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:49:19.496599 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:19.496201 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:49:19.508307 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:19.508286 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:49:19.557950 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:19.557930 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:49:20.668609 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:20.668578 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl"] Sep 16 13:49:20.671275 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:20.668822 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" podUID="411dca39-436d-4b3f-b5d0-ac1b46de461d" containerName="maas-api" containerID="cri-o://5b078daabf91103131a2049ea69651f78aa54874da968adcdc605bd3b23ea842" gracePeriod=30 Sep 16 13:49:20.928450 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:20.928389 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" Sep 16 13:49:20.981473 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:20.981439 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411dca39-436d-4b3f-b5d0-ac1b46de461d-maas-api-tls\") pod \"411dca39-436d-4b3f-b5d0-ac1b46de461d\" (UID: \"411dca39-436d-4b3f-b5d0-ac1b46de461d\") " Sep 16 13:49:20.981645 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:20.981515 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411dca39-436d-4b3f-b5d0-ac1b46de461d-openshift-service-ca\") pod \"411dca39-436d-4b3f-b5d0-ac1b46de461d\" (UID: \"411dca39-436d-4b3f-b5d0-ac1b46de461d\") " Sep 16 13:49:20.981645 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:20.981544 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tmwd\" (UniqueName: \"kubernetes.io/projected/411dca39-436d-4b3f-b5d0-ac1b46de461d-kube-api-access-8tmwd\") pod \"411dca39-436d-4b3f-b5d0-ac1b46de461d\" (UID: \"411dca39-436d-4b3f-b5d0-ac1b46de461d\") " Sep 16 13:49:20.981645 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:20.981620 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/411dca39-436d-4b3f-b5d0-ac1b46de461d-metrics-certs\") pod \"411dca39-436d-4b3f-b5d0-ac1b46de461d\" (UID: \"411dca39-436d-4b3f-b5d0-ac1b46de461d\") " Sep 16 13:49:20.981943 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:20.981912 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/411dca39-436d-4b3f-b5d0-ac1b46de461d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "411dca39-436d-4b3f-b5d0-ac1b46de461d" (UID: "411dca39-436d-4b3f-b5d0-ac1b46de461d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:49:20.982123 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:20.982101 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411dca39-436d-4b3f-b5d0-ac1b46de461d-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:49:20.983906 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:20.983884 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/411dca39-436d-4b3f-b5d0-ac1b46de461d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "411dca39-436d-4b3f-b5d0-ac1b46de461d" (UID: "411dca39-436d-4b3f-b5d0-ac1b46de461d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:49:20.984351 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:20.984333 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/411dca39-436d-4b3f-b5d0-ac1b46de461d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "411dca39-436d-4b3f-b5d0-ac1b46de461d" (UID: "411dca39-436d-4b3f-b5d0-ac1b46de461d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:49:20.984351 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:20.984341 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/411dca39-436d-4b3f-b5d0-ac1b46de461d-kube-api-access-8tmwd" (OuterVolumeSpecName: "kube-api-access-8tmwd") pod "411dca39-436d-4b3f-b5d0-ac1b46de461d" (UID: "411dca39-436d-4b3f-b5d0-ac1b46de461d"). InnerVolumeSpecName "kube-api-access-8tmwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:49:21.083209 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:21.083172 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411dca39-436d-4b3f-b5d0-ac1b46de461d-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:49:21.083209 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:21.083209 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tmwd\" (UniqueName: \"kubernetes.io/projected/411dca39-436d-4b3f-b5d0-ac1b46de461d-kube-api-access-8tmwd\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:49:21.083209 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:21.083222 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/411dca39-436d-4b3f-b5d0-ac1b46de461d-metrics-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:49:21.560963 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:21.560928 2562 generic.go:356] "Generic (PLEG): container finished" podID="411dca39-436d-4b3f-b5d0-ac1b46de461d" containerID="5b078daabf91103131a2049ea69651f78aa54874da968adcdc605bd3b23ea842" exitCode=0 Sep 16 13:49:21.561147 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:21.560977 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" Sep 16 13:49:21.561147 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:21.561014 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" event={"ID":"411dca39-436d-4b3f-b5d0-ac1b46de461d","Type":"ContainerDied","Data":"5b078daabf91103131a2049ea69651f78aa54874da968adcdc605bd3b23ea842"} Sep 16 13:49:21.561147 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:21.561076 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl" event={"ID":"411dca39-436d-4b3f-b5d0-ac1b46de461d","Type":"ContainerDied","Data":"4dafa6e309868c13480113d95cf923f40341865b933aa11752a9f18fe30f9714"} Sep 16 13:49:21.561147 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:21.561104 2562 scope.go:117] "RemoveContainer" containerID="5b078daabf91103131a2049ea69651f78aa54874da968adcdc605bd3b23ea842" Sep 16 13:49:21.571685 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:21.571663 2562 scope.go:117] "RemoveContainer" containerID="5b078daabf91103131a2049ea69651f78aa54874da968adcdc605bd3b23ea842" Sep 16 13:49:21.572000 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:49:21.571976 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b078daabf91103131a2049ea69651f78aa54874da968adcdc605bd3b23ea842\": container with ID starting with 5b078daabf91103131a2049ea69651f78aa54874da968adcdc605bd3b23ea842 not found: ID does not exist" containerID="5b078daabf91103131a2049ea69651f78aa54874da968adcdc605bd3b23ea842" Sep 16 13:49:21.572054 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:21.572013 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b078daabf91103131a2049ea69651f78aa54874da968adcdc605bd3b23ea842"} err="failed to get container status \"5b078daabf91103131a2049ea69651f78aa54874da968adcdc605bd3b23ea842\": rpc error: code = NotFound desc = could not find container \"5b078daabf91103131a2049ea69651f78aa54874da968adcdc605bd3b23ea842\": container with ID starting with 5b078daabf91103131a2049ea69651f78aa54874da968adcdc605bd3b23ea842 not found: ID does not exist" Sep 16 13:49:21.584218 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:21.584190 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl"] Sep 16 13:49:21.589046 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:21.589016 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ab3aa7c0-5cb99668f4-fr7pl"] Sep 16 13:49:22.730010 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:22.729979 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv"] Sep 16 13:49:22.730499 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:22.730187 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" podUID="eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b" containerName="maas-api" containerID="cri-o://da39c65da5c1835c902cfd541e4f27f422df8e90d151d5ce84da4a0176f029ce" gracePeriod=30 Sep 16 13:49:22.776486 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:22.776450 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" podUID="eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.61:8443/health\": dial tcp 10.132.0.61:8443: connect: connection refused" Sep 16 13:49:22.978575 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:22.978544 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" Sep 16 13:49:23.105050 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:23.104973 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-metrics-certs\") pod \"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b\" (UID: \"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b\") " Sep 16 13:49:23.105215 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:23.105103 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-openshift-service-ca\") pod \"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b\" (UID: \"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b\") " Sep 16 13:49:23.105215 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:23.105175 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kt6c7\" (UniqueName: \"kubernetes.io/projected/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-kube-api-access-kt6c7\") pod \"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b\" (UID: \"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b\") " Sep 16 13:49:23.105215 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:23.105203 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-maas-api-tls\") pod \"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b\" (UID: \"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b\") " Sep 16 13:49:23.105541 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:23.105512 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b" (UID: "eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:49:23.107364 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:23.107341 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b" (UID: "eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:49:23.107491 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:23.107430 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-kube-api-access-kt6c7" (OuterVolumeSpecName: "kube-api-access-kt6c7") pod "eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b" (UID: "eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b"). InnerVolumeSpecName "kube-api-access-kt6c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:49:23.108017 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:23.108000 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b" (UID: "eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:49:23.206954 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:23.206925 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:49:23.206954 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:23.206952 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt6c7\" (UniqueName: \"kubernetes.io/projected/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-kube-api-access-kt6c7\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:49:23.207096 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:23.206962 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:49:23.207096 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:23.206971 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b-metrics-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:49:23.414363 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:23.414331 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="411dca39-436d-4b3f-b5d0-ac1b46de461d" path="/var/lib/kubelet/pods/411dca39-436d-4b3f-b5d0-ac1b46de461d/volumes" Sep 16 13:49:23.575354 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:23.575320 2562 generic.go:356] "Generic (PLEG): container finished" podID="eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b" containerID="da39c65da5c1835c902cfd541e4f27f422df8e90d151d5ce84da4a0176f029ce" exitCode=0 Sep 16 13:49:23.575531 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:23.575364 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" Sep 16 13:49:23.575531 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:23.575404 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" event={"ID":"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b","Type":"ContainerDied","Data":"da39c65da5c1835c902cfd541e4f27f422df8e90d151d5ce84da4a0176f029ce"} Sep 16 13:49:23.575531 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:23.575459 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv" event={"ID":"eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b","Type":"ContainerDied","Data":"f0ec7db880521c07ef6a417855cf0a190f7ad34fdfe4e1a9d51533ccbf393ea4"} Sep 16 13:49:23.575531 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:23.575481 2562 scope.go:117] "RemoveContainer" containerID="da39c65da5c1835c902cfd541e4f27f422df8e90d151d5ce84da4a0176f029ce" Sep 16 13:49:23.587464 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:23.587450 2562 scope.go:117] "RemoveContainer" containerID="da39c65da5c1835c902cfd541e4f27f422df8e90d151d5ce84da4a0176f029ce" Sep 16 13:49:23.587784 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:49:23.587763 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da39c65da5c1835c902cfd541e4f27f422df8e90d151d5ce84da4a0176f029ce\": container with ID starting with da39c65da5c1835c902cfd541e4f27f422df8e90d151d5ce84da4a0176f029ce not found: ID does not exist" containerID="da39c65da5c1835c902cfd541e4f27f422df8e90d151d5ce84da4a0176f029ce" Sep 16 13:49:23.587884 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:23.587788 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da39c65da5c1835c902cfd541e4f27f422df8e90d151d5ce84da4a0176f029ce"} err="failed to get container status \"da39c65da5c1835c902cfd541e4f27f422df8e90d151d5ce84da4a0176f029ce\": rpc error: code = NotFound desc = could not find container \"da39c65da5c1835c902cfd541e4f27f422df8e90d151d5ce84da4a0176f029ce\": container with ID starting with da39c65da5c1835c902cfd541e4f27f422df8e90d151d5ce84da4a0176f029ce not found: ID does not exist" Sep 16 13:49:23.601054 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:23.601030 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv"] Sep 16 13:49:23.611696 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:23.611672 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4678b2-6f5d7d666d-7hrvv"] Sep 16 13:49:25.414743 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:25.414714 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b" path="/var/lib/kubelet/pods/eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b/volumes" Sep 16 13:49:49.299257 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:49.299225 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796"] Sep 16 13:49:49.300147 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:49.300125 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="411dca39-436d-4b3f-b5d0-ac1b46de461d" containerName="maas-api" Sep 16 13:49:49.300147 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:49.300145 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="411dca39-436d-4b3f-b5d0-ac1b46de461d" containerName="maas-api" Sep 16 13:49:49.300321 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:49.300178 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b" containerName="maas-api" Sep 16 13:49:49.300321 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:49.300187 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b" containerName="maas-api" Sep 16 13:49:49.300321 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:49.300277 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaaa0a31-f3cf-40ab-b8f8-e6e8c7046e3b" containerName="maas-api" Sep 16 13:49:49.300321 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:49.300293 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="411dca39-436d-4b3f-b5d0-ac1b46de461d" containerName="maas-api" Sep 16 13:49:49.304822 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:49.304802 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" Sep 16 13:49:49.307473 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:49.307450 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-aff7af-serving-cert\"" Sep 16 13:49:49.315788 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:49.315763 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796"] Sep 16 13:49:49.447115 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:49.447083 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/741e39ff-8c9e-46da-b9b2-6ae7e259825f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796\" (UID: \"741e39ff-8c9e-46da-b9b2-6ae7e259825f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" Sep 16 13:49:49.447271 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:49.447126 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/741e39ff-8c9e-46da-b9b2-6ae7e259825f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796\" (UID: \"741e39ff-8c9e-46da-b9b2-6ae7e259825f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" Sep 16 13:49:49.447271 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:49.447197 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/741e39ff-8c9e-46da-b9b2-6ae7e259825f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796\" (UID: \"741e39ff-8c9e-46da-b9b2-6ae7e259825f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" Sep 16 13:49:49.447271 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:49.447267 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmw69\" (UniqueName: \"kubernetes.io/projected/741e39ff-8c9e-46da-b9b2-6ae7e259825f-kube-api-access-tmw69\") pod \"maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796\" (UID: \"741e39ff-8c9e-46da-b9b2-6ae7e259825f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" Sep 16 13:49:49.547829 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:49.547796 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmw69\" (UniqueName: \"kubernetes.io/projected/741e39ff-8c9e-46da-b9b2-6ae7e259825f-kube-api-access-tmw69\") pod \"maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796\" (UID: \"741e39ff-8c9e-46da-b9b2-6ae7e259825f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" Sep 16 13:49:49.547994 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:49.547850 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/741e39ff-8c9e-46da-b9b2-6ae7e259825f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796\" (UID: \"741e39ff-8c9e-46da-b9b2-6ae7e259825f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" Sep 16 13:49:49.547994 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:49.547891 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/741e39ff-8c9e-46da-b9b2-6ae7e259825f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796\" (UID: \"741e39ff-8c9e-46da-b9b2-6ae7e259825f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" Sep 16 13:49:49.547994 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:49.547968 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/741e39ff-8c9e-46da-b9b2-6ae7e259825f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796\" (UID: \"741e39ff-8c9e-46da-b9b2-6ae7e259825f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" Sep 16 13:49:49.547994 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:49:49.547983 2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-serving-cert: secret "maas-api-e2e-shared-a-w2-aff7af-serving-cert" not found Sep 16 13:49:49.548221 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:49:49.548051 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/741e39ff-8c9e-46da-b9b2-6ae7e259825f-maas-api-tls podName:741e39ff-8c9e-46da-b9b2-6ae7e259825f nodeName:}" failed. No retries permitted until 2026-09-16 13:49:50.048037273 +0000 UTC m=+1151.092340217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/741e39ff-8c9e-46da-b9b2-6ae7e259825f-maas-api-tls") pod "maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" (UID: "741e39ff-8c9e-46da-b9b2-6ae7e259825f") : secret "maas-api-e2e-shared-a-w2-aff7af-serving-cert" not found Sep 16 13:49:49.548559 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:49.548533 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/741e39ff-8c9e-46da-b9b2-6ae7e259825f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796\" (UID: \"741e39ff-8c9e-46da-b9b2-6ae7e259825f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" Sep 16 13:49:49.550573 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:49.550527 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/741e39ff-8c9e-46da-b9b2-6ae7e259825f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796\" (UID: \"741e39ff-8c9e-46da-b9b2-6ae7e259825f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" Sep 16 13:49:49.560715 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:49.560695 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmw69\" (UniqueName: \"kubernetes.io/projected/741e39ff-8c9e-46da-b9b2-6ae7e259825f-kube-api-access-tmw69\") pod \"maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796\" (UID: \"741e39ff-8c9e-46da-b9b2-6ae7e259825f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" Sep 16 13:49:50.053860 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:50.053826 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/741e39ff-8c9e-46da-b9b2-6ae7e259825f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796\" (UID: \"741e39ff-8c9e-46da-b9b2-6ae7e259825f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" Sep 16 13:49:50.057042 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:50.057010 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/741e39ff-8c9e-46da-b9b2-6ae7e259825f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796\" (UID: \"741e39ff-8c9e-46da-b9b2-6ae7e259825f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" Sep 16 13:49:50.220067 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:50.220037 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" Sep 16 13:49:50.492540 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:50.492517 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796"] Sep 16 13:49:50.493271 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:49:50.493248 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741e39ff_8c9e_46da_b9b2_6ae7e259825f.slice/crio-fabb6c12279d8b5de3a87407a9d991683341f7062e7cda015a98814bed2ce407 WatchSource:0}: Error finding container fabb6c12279d8b5de3a87407a9d991683341f7062e7cda015a98814bed2ce407: Status 404 returned error can't find the container with id fabb6c12279d8b5de3a87407a9d991683341f7062e7cda015a98814bed2ce407 Sep 16 13:49:50.738554 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:50.738523 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" event={"ID":"741e39ff-8c9e-46da-b9b2-6ae7e259825f","Type":"ContainerStarted","Data":"fabb6c12279d8b5de3a87407a9d991683341f7062e7cda015a98814bed2ce407"} Sep 16 13:49:51.752495 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:51.752453 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" event={"ID":"741e39ff-8c9e-46da-b9b2-6ae7e259825f","Type":"ContainerStarted","Data":"8957e968759aebb45c64fa1437bc91e7bc0fe3c5d86973d90db545dddc2cd614"} Sep 16 13:49:51.752867 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:51.752549 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" Sep 16 13:49:51.770986 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:51.770944 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" podStartSLOduration=2.389159849 podStartE2EDuration="2.770929693s" totalImagesPullingTime="381.769844ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:49:51.769754619 +0000 UTC m=+1152.814057608" watchObservedRunningTime="2026-09-16 13:49:51.770929693 +0000 UTC m=+1152.815232659" Sep 16 13:49:57.763553 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:49:57.763523 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" Sep 16 13:50:16.391897 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:16.391866 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp"] Sep 16 13:50:16.392328 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:16.392086 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp" podUID="0cc0bbd2-504a-47d9-bb6b-320cee929918" containerName="maas-api" containerID="cri-o://33abb7597b15723238c1abdbe82674893e5d2961336c407fef300efa7abd845e" gracePeriod=30 Sep 16 13:50:16.654450 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:16.654327 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp" Sep 16 13:50:16.811535 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:16.811500 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cc0bbd2-504a-47d9-bb6b-320cee929918-maas-api-tls\") pod \"0cc0bbd2-504a-47d9-bb6b-320cee929918\" (UID: \"0cc0bbd2-504a-47d9-bb6b-320cee929918\") " Sep 16 13:50:16.811535 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:16.811554 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc0bbd2-504a-47d9-bb6b-320cee929918-openshift-service-ca\") pod \"0cc0bbd2-504a-47d9-bb6b-320cee929918\" (UID: \"0cc0bbd2-504a-47d9-bb6b-320cee929918\") " Sep 16 13:50:16.811796 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:16.811600 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvt5g\" (UniqueName: \"kubernetes.io/projected/0cc0bbd2-504a-47d9-bb6b-320cee929918-kube-api-access-fvt5g\") pod \"0cc0bbd2-504a-47d9-bb6b-320cee929918\" (UID: \"0cc0bbd2-504a-47d9-bb6b-320cee929918\") " Sep 16 13:50:16.811796 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:16.811683 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cc0bbd2-504a-47d9-bb6b-320cee929918-metrics-certs\") pod \"0cc0bbd2-504a-47d9-bb6b-320cee929918\" (UID: \"0cc0bbd2-504a-47d9-bb6b-320cee929918\") " Sep 16 13:50:16.811971 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:16.811948 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc0bbd2-504a-47d9-bb6b-320cee929918-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0cc0bbd2-504a-47d9-bb6b-320cee929918" (UID: "0cc0bbd2-504a-47d9-bb6b-320cee929918"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:50:16.813864 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:16.813831 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cc0bbd2-504a-47d9-bb6b-320cee929918-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0cc0bbd2-504a-47d9-bb6b-320cee929918" (UID: "0cc0bbd2-504a-47d9-bb6b-320cee929918"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:50:16.813864 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:16.813847 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cc0bbd2-504a-47d9-bb6b-320cee929918-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0cc0bbd2-504a-47d9-bb6b-320cee929918" (UID: "0cc0bbd2-504a-47d9-bb6b-320cee929918"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:50:16.813999 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:16.813884 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc0bbd2-504a-47d9-bb6b-320cee929918-kube-api-access-fvt5g" (OuterVolumeSpecName: "kube-api-access-fvt5g") pod "0cc0bbd2-504a-47d9-bb6b-320cee929918" (UID: "0cc0bbd2-504a-47d9-bb6b-320cee929918"). InnerVolumeSpecName "kube-api-access-fvt5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:50:16.912739 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:16.912712 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cc0bbd2-504a-47d9-bb6b-320cee929918-metrics-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:50:16.912903 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:16.912768 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cc0bbd2-504a-47d9-bb6b-320cee929918-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:50:16.912903 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:16.912783 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc0bbd2-504a-47d9-bb6b-320cee929918-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:50:16.912903 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:16.912797 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvt5g\" (UniqueName: \"kubernetes.io/projected/0cc0bbd2-504a-47d9-bb6b-320cee929918-kube-api-access-fvt5g\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:50:16.913180 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:16.913159 2562 generic.go:356] "Generic (PLEG): container finished" podID="0cc0bbd2-504a-47d9-bb6b-320cee929918" containerID="33abb7597b15723238c1abdbe82674893e5d2961336c407fef300efa7abd845e" exitCode=0 Sep 16 13:50:16.913240 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:16.913223 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp" Sep 16 13:50:16.913290 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:16.913247 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp" event={"ID":"0cc0bbd2-504a-47d9-bb6b-320cee929918","Type":"ContainerDied","Data":"33abb7597b15723238c1abdbe82674893e5d2961336c407fef300efa7abd845e"} Sep 16 13:50:16.913343 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:16.913320 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp" event={"ID":"0cc0bbd2-504a-47d9-bb6b-320cee929918","Type":"ContainerDied","Data":"7fb8245ffb1cbd03370ab7bcea5985e9e3c2de0f95b8ed496b71d065088b9626"} Sep 16 13:50:16.913428 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:16.913353 2562 scope.go:117] "RemoveContainer" containerID="33abb7597b15723238c1abdbe82674893e5d2961336c407fef300efa7abd845e" Sep 16 13:50:16.925002 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:16.924985 2562 scope.go:117] "RemoveContainer" containerID="33abb7597b15723238c1abdbe82674893e5d2961336c407fef300efa7abd845e" Sep 16 13:50:16.925316 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:50:16.925292 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33abb7597b15723238c1abdbe82674893e5d2961336c407fef300efa7abd845e\": container with ID starting with 33abb7597b15723238c1abdbe82674893e5d2961336c407fef300efa7abd845e not found: ID does not exist" containerID="33abb7597b15723238c1abdbe82674893e5d2961336c407fef300efa7abd845e" Sep 16 13:50:16.925449 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:16.925324 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33abb7597b15723238c1abdbe82674893e5d2961336c407fef300efa7abd845e"} err="failed to get container status \"33abb7597b15723238c1abdbe82674893e5d2961336c407fef300efa7abd845e\": rpc error: code = NotFound desc = could not find container \"33abb7597b15723238c1abdbe82674893e5d2961336c407fef300efa7abd845e\": container with ID starting with 33abb7597b15723238c1abdbe82674893e5d2961336c407fef300efa7abd845e not found: ID does not exist" Sep 16 13:50:16.967871 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:16.967849 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp"] Sep 16 13:50:17.000547 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:17.000525 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aca190-776685b4cf-55slp"] Sep 16 13:50:17.413826 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:17.413798 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cc0bbd2-504a-47d9-bb6b-320cee929918" path="/var/lib/kubelet/pods/0cc0bbd2-504a-47d9-bb6b-320cee929918/volumes" Sep 16 13:50:20.611606 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:20.611573 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx"] Sep 16 13:50:20.612280 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:20.612260 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cc0bbd2-504a-47d9-bb6b-320cee929918" containerName="maas-api" Sep 16 13:50:20.612322 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:20.612285 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cc0bbd2-504a-47d9-bb6b-320cee929918" containerName="maas-api" Sep 16 13:50:20.612446 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:20.612432 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cc0bbd2-504a-47d9-bb6b-320cee929918" containerName="maas-api" Sep 16 13:50:20.617109 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:20.617090 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx" Sep 16 13:50:20.619500 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:20.619475 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-f2afd5-serving-cert\"" Sep 16 13:50:20.624523 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:20.624505 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx"] Sep 16 13:50:20.752421 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:20.752359 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bae6be5-baf4-4e43-960a-24c42e43a1ad-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx\" (UID: \"5bae6be5-baf4-4e43-960a-24c42e43a1ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx" Sep 16 13:50:20.752613 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:20.752484 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bae6be5-baf4-4e43-960a-24c42e43a1ad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx\" (UID: \"5bae6be5-baf4-4e43-960a-24c42e43a1ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx" Sep 16 13:50:20.752613 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:20.752555 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bae6be5-baf4-4e43-960a-24c42e43a1ad-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx\" (UID: \"5bae6be5-baf4-4e43-960a-24c42e43a1ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx" Sep 16 13:50:20.752707 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:20.752636 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlhjc\" (UniqueName: \"kubernetes.io/projected/5bae6be5-baf4-4e43-960a-24c42e43a1ad-kube-api-access-vlhjc\") pod \"maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx\" (UID: \"5bae6be5-baf4-4e43-960a-24c42e43a1ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx" Sep 16 13:50:20.853782 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:20.853754 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bae6be5-baf4-4e43-960a-24c42e43a1ad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx\" (UID: \"5bae6be5-baf4-4e43-960a-24c42e43a1ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx" Sep 16 13:50:20.853944 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:20.853808 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bae6be5-baf4-4e43-960a-24c42e43a1ad-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx\" (UID: \"5bae6be5-baf4-4e43-960a-24c42e43a1ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx" Sep 16 13:50:20.853944 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:20.853850 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlhjc\" (UniqueName: \"kubernetes.io/projected/5bae6be5-baf4-4e43-960a-24c42e43a1ad-kube-api-access-vlhjc\") pod \"maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx\" (UID: \"5bae6be5-baf4-4e43-960a-24c42e43a1ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx" Sep 16 13:50:20.853944 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:20.853890 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bae6be5-baf4-4e43-960a-24c42e43a1ad-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx\" (UID: \"5bae6be5-baf4-4e43-960a-24c42e43a1ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx" Sep 16 13:50:20.854539 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:20.854519 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bae6be5-baf4-4e43-960a-24c42e43a1ad-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx\" (UID: \"5bae6be5-baf4-4e43-960a-24c42e43a1ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx" Sep 16 13:50:20.856473 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:20.856448 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bae6be5-baf4-4e43-960a-24c42e43a1ad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx\" (UID: \"5bae6be5-baf4-4e43-960a-24c42e43a1ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx" Sep 16 13:50:20.856561 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:20.856504 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bae6be5-baf4-4e43-960a-24c42e43a1ad-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx\" (UID: \"5bae6be5-baf4-4e43-960a-24c42e43a1ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx" Sep 16 13:50:20.865751 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:20.865702 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlhjc\" (UniqueName: \"kubernetes.io/projected/5bae6be5-baf4-4e43-960a-24c42e43a1ad-kube-api-access-vlhjc\") pod \"maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx\" (UID: \"5bae6be5-baf4-4e43-960a-24c42e43a1ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx" Sep 16 13:50:20.930517 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:20.930487 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx" Sep 16 13:50:21.240814 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:21.240787 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx"] Sep 16 13:50:21.241460 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:50:21.241427 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bae6be5_baf4_4e43_960a_24c42e43a1ad.slice/crio-f6e5fac6b0faa3ebb7a4febdbcdca8ffe41e7122670708a99b0d98575bea040d WatchSource:0}: Error finding container f6e5fac6b0faa3ebb7a4febdbcdca8ffe41e7122670708a99b0d98575bea040d: Status 404 returned error can't find the container with id f6e5fac6b0faa3ebb7a4febdbcdca8ffe41e7122670708a99b0d98575bea040d Sep 16 13:50:21.949783 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:21.949752 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx" event={"ID":"5bae6be5-baf4-4e43-960a-24c42e43a1ad","Type":"ContainerStarted","Data":"79a3d3d1f10054985afb9a6c890061e4927165d37ba5b8770e63c417e6a1a1e9"} Sep 16 13:50:21.950146 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:21.949804 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx" event={"ID":"5bae6be5-baf4-4e43-960a-24c42e43a1ad","Type":"ContainerStarted","Data":"f6e5fac6b0faa3ebb7a4febdbcdca8ffe41e7122670708a99b0d98575bea040d"} Sep 16 13:50:21.950146 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:21.949831 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx" Sep 16 13:50:21.972483 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:21.972426 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx" podStartSLOduration=1.5942912329999999 podStartE2EDuration="1.972406976s" totalImagesPullingTime="378.115743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:50:21.968364484 +0000 UTC m=+1183.012667473" watchObservedRunningTime="2026-09-16 13:50:21.972406976 +0000 UTC m=+1183.016709942" Sep 16 13:50:23.937878 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:23.937848 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv"] Sep 16 13:50:23.938320 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:23.938030 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" podUID="45e60603-fc1d-4b9d-a9ac-492b7be491e7" containerName="istio-proxy" containerID="cri-o://be4ee03f11d3624c3db9f61489f7048013dfc5a2292995010238e24ec456144d" gracePeriod=30 Sep 16 13:50:27.961268 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:27.961226 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx" Sep 16 13:50:29.187076 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.187052 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:50:29.237654 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.235155 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istio-podinfo\") pod \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " Sep 16 13:50:29.237654 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.235608 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-workload-certs\") pod \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " Sep 16 13:50:29.237654 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.235644 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istiod-ca-cert\") pod \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " Sep 16 13:50:29.237654 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.235667 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6grvc\" (UniqueName: \"kubernetes.io/projected/45e60603-fc1d-4b9d-a9ac-492b7be491e7-kube-api-access-6grvc\") pod \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " Sep 16 13:50:29.237654 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.235716 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istio-token\") pod \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " Sep 16 13:50:29.237654 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.235745 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-credential-socket\") pod \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " Sep 16 13:50:29.237654 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.235866 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istio-envoy\") pod \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " Sep 16 13:50:29.237654 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.235895 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-workload-socket\") pod \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " Sep 16 13:50:29.237654 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.235935 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istio-data\") pod \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\" (UID: \"45e60603-fc1d-4b9d-a9ac-492b7be491e7\") " Sep 16 13:50:29.237654 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.236543 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istio-data" (OuterVolumeSpecName: "istio-data") pod "45e60603-fc1d-4b9d-a9ac-492b7be491e7" (UID: "45e60603-fc1d-4b9d-a9ac-492b7be491e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:50:29.237654 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.236727 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "45e60603-fc1d-4b9d-a9ac-492b7be491e7" (UID: "45e60603-fc1d-4b9d-a9ac-492b7be491e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:50:29.237654 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.236989 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "45e60603-fc1d-4b9d-a9ac-492b7be491e7" (UID: "45e60603-fc1d-4b9d-a9ac-492b7be491e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:50:29.237654 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.237263 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "45e60603-fc1d-4b9d-a9ac-492b7be491e7" (UID: "45e60603-fc1d-4b9d-a9ac-492b7be491e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:50:29.237654 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.237352 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "45e60603-fc1d-4b9d-a9ac-492b7be491e7" (UID: "45e60603-fc1d-4b9d-a9ac-492b7be491e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:50:29.240170 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.240131 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "45e60603-fc1d-4b9d-a9ac-492b7be491e7" (UID: "45e60603-fc1d-4b9d-a9ac-492b7be491e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:50:29.241031 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.240996 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istio-token" (OuterVolumeSpecName: "istio-token") pod "45e60603-fc1d-4b9d-a9ac-492b7be491e7" (UID: "45e60603-fc1d-4b9d-a9ac-492b7be491e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:50:29.241151 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.241110 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "45e60603-fc1d-4b9d-a9ac-492b7be491e7" (UID: "45e60603-fc1d-4b9d-a9ac-492b7be491e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 13:50:29.241280 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.241261 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e60603-fc1d-4b9d-a9ac-492b7be491e7-kube-api-access-6grvc" (OuterVolumeSpecName: "kube-api-access-6grvc") pod "45e60603-fc1d-4b9d-a9ac-492b7be491e7" (UID: "45e60603-fc1d-4b9d-a9ac-492b7be491e7"). InnerVolumeSpecName "kube-api-access-6grvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:50:29.336951 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.336928 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istio-envoy\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:50:29.336951 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.336952 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-workload-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:50:29.337101 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.336960 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istio-data\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:50:29.337101 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.336971 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istio-podinfo\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:50:29.337101 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.336978 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-workload-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:50:29.337101 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.336985 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istiod-ca-cert\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:50:29.337101 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.336993 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6grvc\" (UniqueName: \"kubernetes.io/projected/45e60603-fc1d-4b9d-a9ac-492b7be491e7-kube-api-access-6grvc\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:50:29.337101 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.337001 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e60603-fc1d-4b9d-a9ac-492b7be491e7-istio-token\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:50:29.337101 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:29.337009 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e60603-fc1d-4b9d-a9ac-492b7be491e7-credential-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:50:30.003309 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:30.003275 2562 generic.go:356] "Generic (PLEG): container finished" podID="45e60603-fc1d-4b9d-a9ac-492b7be491e7" containerID="be4ee03f11d3624c3db9f61489f7048013dfc5a2292995010238e24ec456144d" exitCode=0 Sep 16 13:50:30.003493 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:30.003326 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" Sep 16 13:50:30.003493 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:30.003356 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" event={"ID":"45e60603-fc1d-4b9d-a9ac-492b7be491e7","Type":"ContainerDied","Data":"be4ee03f11d3624c3db9f61489f7048013dfc5a2292995010238e24ec456144d"} Sep 16 13:50:30.003493 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:30.003428 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv" event={"ID":"45e60603-fc1d-4b9d-a9ac-492b7be491e7","Type":"ContainerDied","Data":"3d2065fc63715cd0d4226f51aca7ded4a4bdb439dcb39cba821cc2e40c034880"} Sep 16 13:50:30.003493 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:30.003451 2562 scope.go:117] "RemoveContainer" containerID="be4ee03f11d3624c3db9f61489f7048013dfc5a2292995010238e24ec456144d" Sep 16 13:50:30.012771 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:30.012751 2562 scope.go:117] "RemoveContainer" containerID="64c050d84c74b3ae0a12a88ea0b1071df67eef24d6d6966962d019a5f3f84381" Sep 16 13:50:30.021259 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:30.021240 2562 scope.go:117] "RemoveContainer" containerID="be4ee03f11d3624c3db9f61489f7048013dfc5a2292995010238e24ec456144d" Sep 16 13:50:30.021965 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:50:30.021926 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4ee03f11d3624c3db9f61489f7048013dfc5a2292995010238e24ec456144d\": container with ID starting with be4ee03f11d3624c3db9f61489f7048013dfc5a2292995010238e24ec456144d not found: ID does not exist" containerID="be4ee03f11d3624c3db9f61489f7048013dfc5a2292995010238e24ec456144d" Sep 16 13:50:30.022089 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:30.021974 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4ee03f11d3624c3db9f61489f7048013dfc5a2292995010238e24ec456144d"} err="failed to get container status \"be4ee03f11d3624c3db9f61489f7048013dfc5a2292995010238e24ec456144d\": rpc error: code = NotFound desc = could not find container \"be4ee03f11d3624c3db9f61489f7048013dfc5a2292995010238e24ec456144d\": container with ID starting with be4ee03f11d3624c3db9f61489f7048013dfc5a2292995010238e24ec456144d not found: ID does not exist" Sep 16 13:50:30.022089 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:30.021992 2562 scope.go:117] "RemoveContainer" containerID="64c050d84c74b3ae0a12a88ea0b1071df67eef24d6d6966962d019a5f3f84381" Sep 16 13:50:30.022583 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:50:30.022563 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64c050d84c74b3ae0a12a88ea0b1071df67eef24d6d6966962d019a5f3f84381\": container with ID starting with 64c050d84c74b3ae0a12a88ea0b1071df67eef24d6d6966962d019a5f3f84381 not found: ID does not exist" containerID="64c050d84c74b3ae0a12a88ea0b1071df67eef24d6d6966962d019a5f3f84381" Sep 16 13:50:30.022662 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:30.022594 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64c050d84c74b3ae0a12a88ea0b1071df67eef24d6d6966962d019a5f3f84381"} err="failed to get container status \"64c050d84c74b3ae0a12a88ea0b1071df67eef24d6d6966962d019a5f3f84381\": rpc error: code = NotFound desc = could not find container \"64c050d84c74b3ae0a12a88ea0b1071df67eef24d6d6966962d019a5f3f84381\": container with ID starting with 64c050d84c74b3ae0a12a88ea0b1071df67eef24d6d6966962d019a5f3f84381 not found: ID does not exist" Sep 16 13:50:30.024976 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:30.024937 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv"] Sep 16 13:50:30.029143 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:30.029123 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-aca190-openshift-default-6577fd8bdd-jg6tv"] Sep 16 13:50:31.414574 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:31.414540 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45e60603-fc1d-4b9d-a9ac-492b7be491e7" path="/var/lib/kubelet/pods/45e60603-fc1d-4b9d-a9ac-492b7be491e7/volumes" Sep 16 13:50:34.922610 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:34.922570 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf"] Sep 16 13:50:34.923068 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:34.922835 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" podUID="37786cdb-af61-4416-8454-01255ca46b7f" containerName="maas-api" containerID="cri-o://3619870afe6ab30ec9cae228458f392d05af930a097ce46a4bbdba111201a400" gracePeriod=30 Sep 16 13:50:35.175686 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:35.175627 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" Sep 16 13:50:35.293580 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:35.293552 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37786cdb-af61-4416-8454-01255ca46b7f-openshift-service-ca\") pod \"37786cdb-af61-4416-8454-01255ca46b7f\" (UID: \"37786cdb-af61-4416-8454-01255ca46b7f\") " Sep 16 13:50:35.293745 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:35.293648 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qdth\" (UniqueName: \"kubernetes.io/projected/37786cdb-af61-4416-8454-01255ca46b7f-kube-api-access-9qdth\") pod \"37786cdb-af61-4416-8454-01255ca46b7f\" (UID: \"37786cdb-af61-4416-8454-01255ca46b7f\") " Sep 16 13:50:35.293745 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:35.293689 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37786cdb-af61-4416-8454-01255ca46b7f-metrics-certs\") pod \"37786cdb-af61-4416-8454-01255ca46b7f\" (UID: \"37786cdb-af61-4416-8454-01255ca46b7f\") " Sep 16 13:50:35.293745 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:35.293728 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37786cdb-af61-4416-8454-01255ca46b7f-maas-api-tls\") pod \"37786cdb-af61-4416-8454-01255ca46b7f\" (UID: \"37786cdb-af61-4416-8454-01255ca46b7f\") " Sep 16 13:50:35.293953 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:35.293921 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37786cdb-af61-4416-8454-01255ca46b7f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "37786cdb-af61-4416-8454-01255ca46b7f" (UID: "37786cdb-af61-4416-8454-01255ca46b7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:50:35.294084 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:35.294056 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37786cdb-af61-4416-8454-01255ca46b7f-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:50:35.295930 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:35.295903 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37786cdb-af61-4416-8454-01255ca46b7f-kube-api-access-9qdth" (OuterVolumeSpecName: "kube-api-access-9qdth") pod "37786cdb-af61-4416-8454-01255ca46b7f" (UID: "37786cdb-af61-4416-8454-01255ca46b7f"). InnerVolumeSpecName "kube-api-access-9qdth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:50:35.295930 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:35.295919 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37786cdb-af61-4416-8454-01255ca46b7f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "37786cdb-af61-4416-8454-01255ca46b7f" (UID: "37786cdb-af61-4416-8454-01255ca46b7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:50:35.296062 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:35.296039 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37786cdb-af61-4416-8454-01255ca46b7f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "37786cdb-af61-4416-8454-01255ca46b7f" (UID: "37786cdb-af61-4416-8454-01255ca46b7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:50:35.395494 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:35.395464 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qdth\" (UniqueName: \"kubernetes.io/projected/37786cdb-af61-4416-8454-01255ca46b7f-kube-api-access-9qdth\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:50:35.395494 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:35.395490 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37786cdb-af61-4416-8454-01255ca46b7f-metrics-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:50:35.395494 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:35.395499 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37786cdb-af61-4416-8454-01255ca46b7f-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:50:36.043056 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:36.043017 2562 generic.go:356] "Generic (PLEG): container finished" podID="37786cdb-af61-4416-8454-01255ca46b7f" containerID="3619870afe6ab30ec9cae228458f392d05af930a097ce46a4bbdba111201a400" exitCode=0 Sep 16 13:50:36.043493 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:36.043061 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" Sep 16 13:50:36.043493 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:36.043101 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" event={"ID":"37786cdb-af61-4416-8454-01255ca46b7f","Type":"ContainerDied","Data":"3619870afe6ab30ec9cae228458f392d05af930a097ce46a4bbdba111201a400"} Sep 16 13:50:36.043493 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:36.043155 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf" event={"ID":"37786cdb-af61-4416-8454-01255ca46b7f","Type":"ContainerDied","Data":"84992536b221604ef7033c9001707cdd6e205a69a49579a35091c472f9a947e6"} Sep 16 13:50:36.043493 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:36.043180 2562 scope.go:117] "RemoveContainer" containerID="3619870afe6ab30ec9cae228458f392d05af930a097ce46a4bbdba111201a400" Sep 16 13:50:36.052898 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:36.052884 2562 scope.go:117] "RemoveContainer" containerID="3619870afe6ab30ec9cae228458f392d05af930a097ce46a4bbdba111201a400" Sep 16 13:50:36.053138 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:50:36.053120 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3619870afe6ab30ec9cae228458f392d05af930a097ce46a4bbdba111201a400\": container with ID starting with 3619870afe6ab30ec9cae228458f392d05af930a097ce46a4bbdba111201a400 not found: ID does not exist" containerID="3619870afe6ab30ec9cae228458f392d05af930a097ce46a4bbdba111201a400" Sep 16 13:50:36.053207 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:36.053146 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3619870afe6ab30ec9cae228458f392d05af930a097ce46a4bbdba111201a400"} err="failed to get container status \"3619870afe6ab30ec9cae228458f392d05af930a097ce46a4bbdba111201a400\": rpc error: code = NotFound desc = could not find container \"3619870afe6ab30ec9cae228458f392d05af930a097ce46a4bbdba111201a400\": container with ID starting with 3619870afe6ab30ec9cae228458f392d05af930a097ce46a4bbdba111201a400 not found: ID does not exist" Sep 16 13:50:36.066281 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:36.066258 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf"] Sep 16 13:50:36.071294 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:36.071270 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1e5db0-5fc547c4d-jtqhf"] Sep 16 13:50:37.414475 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:37.414441 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37786cdb-af61-4416-8454-01255ca46b7f" path="/var/lib/kubelet/pods/37786cdb-af61-4416-8454-01255ca46b7f/volumes" Sep 16 13:50:39.486419 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:39.486389 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4xmxv_c7828efb-6be7-4f7b-986f-645a2370163a/console-operator/1.log" Sep 16 13:50:39.490042 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:39.490018 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4xmxv_c7828efb-6be7-4f7b-986f-645a2370163a/console-operator/1.log" Sep 16 13:50:40.783335 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:40.783303 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7309-ffee2f"] Sep 16 13:50:40.783770 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:40.783754 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37786cdb-af61-4416-8454-01255ca46b7f" containerName="maas-api" Sep 16 13:50:40.783813 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:40.783772 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="37786cdb-af61-4416-8454-01255ca46b7f" containerName="maas-api" Sep 16 13:50:40.783813 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:40.783793 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45e60603-fc1d-4b9d-a9ac-492b7be491e7" containerName="istio-proxy" Sep 16 13:50:40.783813 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:40.783798 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="45e60603-fc1d-4b9d-a9ac-492b7be491e7" containerName="istio-proxy" Sep 16 13:50:40.783906 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:40.783865 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="37786cdb-af61-4416-8454-01255ca46b7f" containerName="maas-api" Sep 16 13:50:40.783906 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:40.783874 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="45e60603-fc1d-4b9d-a9ac-492b7be491e7" containerName="istio-proxy" Sep 16 13:50:40.783906 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:40.783882 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="45e60603-fc1d-4b9d-a9ac-492b7be491e7" containerName="istio-proxy" Sep 16 13:50:40.788467 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:40.788441 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7309-ffee2f" Sep 16 13:50:40.791367 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:40.791342 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2kqls\"" Sep 16 13:50:40.792007 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:40.791984 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7309-ffee2f"] Sep 16 13:50:40.847116 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:40.847087 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slq8d\" (UniqueName: \"kubernetes.io/projected/f3cd9ddc-afa6-4746-833c-338d7053546b-kube-api-access-slq8d\") pod \"test-curl-7309-ffee2f\" (UID: \"f3cd9ddc-afa6-4746-833c-338d7053546b\") " pod="opendatahub/test-curl-7309-ffee2f" Sep 16 13:50:40.947940 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:40.947907 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slq8d\" (UniqueName: \"kubernetes.io/projected/f3cd9ddc-afa6-4746-833c-338d7053546b-kube-api-access-slq8d\") pod \"test-curl-7309-ffee2f\" (UID: \"f3cd9ddc-afa6-4746-833c-338d7053546b\") " pod="opendatahub/test-curl-7309-ffee2f" Sep 16 13:50:40.956574 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:40.956552 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slq8d\" (UniqueName: \"kubernetes.io/projected/f3cd9ddc-afa6-4746-833c-338d7053546b-kube-api-access-slq8d\") pod \"test-curl-7309-ffee2f\" (UID: \"f3cd9ddc-afa6-4746-833c-338d7053546b\") " pod="opendatahub/test-curl-7309-ffee2f" Sep 16 13:50:41.101029 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:41.100947 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7309-ffee2f" Sep 16 13:50:41.371030 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:41.371000 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7309-ffee2f"] Sep 16 13:50:41.372355 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:50:41.372330 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3cd9ddc_afa6_4746_833c_338d7053546b.slice/crio-16c2d08f3fa20ea2b39b2cb464416a153e41934b0542fa6bdcaccf7a73eef5f3 WatchSource:0}: Error finding container 16c2d08f3fa20ea2b39b2cb464416a153e41934b0542fa6bdcaccf7a73eef5f3: Status 404 returned error can't find the container with id 16c2d08f3fa20ea2b39b2cb464416a153e41934b0542fa6bdcaccf7a73eef5f3 Sep 16 13:50:42.080578 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:42.080541 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7309-ffee2f" event={"ID":"f3cd9ddc-afa6-4746-833c-338d7053546b","Type":"ContainerStarted","Data":"16c2d08f3fa20ea2b39b2cb464416a153e41934b0542fa6bdcaccf7a73eef5f3"} Sep 16 13:50:44.093879 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:44.093839 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7309-ffee2f" event={"ID":"f3cd9ddc-afa6-4746-833c-338d7053546b","Type":"ContainerStarted","Data":"bf5f7382f87c3a8f75d832f15f104cfc0b0fe51d6c31b7295953aa718757dc7f"} Sep 16 13:50:44.112973 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:44.112929 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7309-ffee2f" podStartSLOduration=1.9829824459999998 podStartE2EDuration="4.112915293s" totalImagesPullingTime="2.129932847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:50:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:50:44.11233746 +0000 UTC m=+1205.156640427" watchObservedRunningTime="2026-09-16 13:50:44.112915293 +0000 UTC m=+1205.157218258" Sep 16 13:50:44.819138 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:44.819106 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7309-ffee2f"] Sep 16 13:50:44.827614 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:44.827583 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7309-ffee2f"] Sep 16 13:50:44.946203 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:44.946172 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7309-502317"] Sep 16 13:50:44.946650 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:44.946637 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45e60603-fc1d-4b9d-a9ac-492b7be491e7" containerName="istio-proxy" Sep 16 13:50:44.946703 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:44.946652 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="45e60603-fc1d-4b9d-a9ac-492b7be491e7" containerName="istio-proxy" Sep 16 13:50:44.946703 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:44.946670 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3cd9ddc-afa6-4746-833c-338d7053546b" containerName="test-curl-7309-ffee2f" Sep 16 13:50:44.946703 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:44.946675 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3cd9ddc-afa6-4746-833c-338d7053546b" containerName="test-curl-7309-ffee2f" Sep 16 13:50:44.946801 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:44.946742 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3cd9ddc-afa6-4746-833c-338d7053546b" containerName="test-curl-7309-ffee2f" Sep 16 13:50:44.949793 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:44.949777 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7309-502317" Sep 16 13:50:44.960345 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:44.960313 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7309-502317"] Sep 16 13:50:45.091627 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:45.091600 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkx6s\" (UniqueName: \"kubernetes.io/projected/8cabcf79-57f6-4e77-972e-df4585a6babd-kube-api-access-dkx6s\") pod \"test-curl-7309-502317\" (UID: \"8cabcf79-57f6-4e77-972e-df4585a6babd\") " pod="opendatahub/test-curl-7309-502317" Sep 16 13:50:45.099607 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:50:45.099587 2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7309-ffee2f" container="test-curl-7309-ffee2f" Sep 16 13:50:45.192193 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:45.192173 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkx6s\" (UniqueName: \"kubernetes.io/projected/8cabcf79-57f6-4e77-972e-df4585a6babd-kube-api-access-dkx6s\") pod \"test-curl-7309-502317\" (UID: \"8cabcf79-57f6-4e77-972e-df4585a6babd\") " pod="opendatahub/test-curl-7309-502317" Sep 16 13:50:45.201452 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:45.201395 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkx6s\" (UniqueName: \"kubernetes.io/projected/8cabcf79-57f6-4e77-972e-df4585a6babd-kube-api-access-dkx6s\") pod \"test-curl-7309-502317\" (UID: \"8cabcf79-57f6-4e77-972e-df4585a6babd\") " pod="opendatahub/test-curl-7309-502317" Sep 16 13:50:45.259585 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:45.259554 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7309-502317" Sep 16 13:50:45.543099 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:50:45.543069 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cabcf79_57f6_4e77_972e_df4585a6babd.slice/crio-e470cb808520f2b18846ee53d954dffe0401a800b90a8867e5ee1a729dde8210 WatchSource:0}: Error finding container e470cb808520f2b18846ee53d954dffe0401a800b90a8867e5ee1a729dde8210: Status 404 returned error can't find the container with id e470cb808520f2b18846ee53d954dffe0401a800b90a8867e5ee1a729dde8210 Sep 16 13:50:45.543099 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:45.543073 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7309-502317"] Sep 16 13:50:46.106547 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:46.106506 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7309-502317" event={"ID":"8cabcf79-57f6-4e77-972e-df4585a6babd","Type":"ContainerStarted","Data":"22bb6b916813dec5892aad93d19c79204e2d27b7f4396f1bd61f550d8b4c6e7d"} Sep 16 13:50:46.106547 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:46.106544 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7309-502317" event={"ID":"8cabcf79-57f6-4e77-972e-df4585a6babd","Type":"ContainerStarted","Data":"e470cb808520f2b18846ee53d954dffe0401a800b90a8867e5ee1a729dde8210"} Sep 16 13:50:46.107032 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:46.106541 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7309-ffee2f" podUID="f3cd9ddc-afa6-4746-833c-338d7053546b" containerName="test-curl-7309-ffee2f" containerID="cri-o://bf5f7382f87c3a8f75d832f15f104cfc0b0fe51d6c31b7295953aa718757dc7f" gracePeriod=2 Sep 16 13:50:46.154287 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:46.154236 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7309-502317" podStartSLOduration=2.154222758 podStartE2EDuration="2.154222758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:50:46.151398264 +0000 UTC m=+1207.195701226" watchObservedRunningTime="2026-09-16 13:50:46.154222758 +0000 UTC m=+1207.198525715" Sep 16 13:50:46.982852 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:46.982778 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7309-502317"] Sep 16 13:50:46.988405 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:46.988365 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7309-502317"] Sep 16 13:50:47.112525 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:50:47.112492 2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7309-502317" container="test-curl-7309-502317" Sep 16 13:50:47.116582 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:47.116562 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7309-9d65b0"] Sep 16 13:50:47.116977 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:47.116965 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cabcf79-57f6-4e77-972e-df4585a6babd" containerName="test-curl-7309-502317" Sep 16 13:50:47.117019 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:47.116980 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cabcf79-57f6-4e77-972e-df4585a6babd" containerName="test-curl-7309-502317" Sep 16 13:50:47.117094 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:47.117062 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cabcf79-57f6-4e77-972e-df4585a6babd" containerName="test-curl-7309-502317" Sep 16 13:50:47.120289 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:47.120275 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7309-9d65b0" Sep 16 13:50:47.154044 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:47.154018 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7309-9d65b0"] Sep 16 13:50:47.211213 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:47.211187 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwvnk\" (UniqueName: \"kubernetes.io/projected/a599570b-1f08-4185-8e81-dcd3a7d37c7b-kube-api-access-lwvnk\") pod \"test-curl-7309-9d65b0\" (UID: \"a599570b-1f08-4185-8e81-dcd3a7d37c7b\") " pod="opendatahub/test-curl-7309-9d65b0" Sep 16 13:50:47.312513 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:47.312429 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwvnk\" (UniqueName: \"kubernetes.io/projected/a599570b-1f08-4185-8e81-dcd3a7d37c7b-kube-api-access-lwvnk\") pod \"test-curl-7309-9d65b0\" (UID: \"a599570b-1f08-4185-8e81-dcd3a7d37c7b\") " pod="opendatahub/test-curl-7309-9d65b0" Sep 16 13:50:47.332602 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:47.332580 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwvnk\" (UniqueName: \"kubernetes.io/projected/a599570b-1f08-4185-8e81-dcd3a7d37c7b-kube-api-access-lwvnk\") pod \"test-curl-7309-9d65b0\" (UID: \"a599570b-1f08-4185-8e81-dcd3a7d37c7b\") " pod="opendatahub/test-curl-7309-9d65b0" Sep 16 13:50:47.430309 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:47.430283 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7309-9d65b0" Sep 16 13:50:47.728005 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:47.727979 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7309-9d65b0"] Sep 16 13:50:47.728875 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:50:47.728851 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda599570b_1f08_4185_8e81_dcd3a7d37c7b.slice/crio-6809cdb3a2502f08ce24d2634647681c59072cde9be000f6f19b51cdd1b190c5 WatchSource:0}: Error finding container 6809cdb3a2502f08ce24d2634647681c59072cde9be000f6f19b51cdd1b190c5: Status 404 returned error can't find the container with id 6809cdb3a2502f08ce24d2634647681c59072cde9be000f6f19b51cdd1b190c5 Sep 16 13:50:48.120998 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:48.120968 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7309-9d65b0" event={"ID":"a599570b-1f08-4185-8e81-dcd3a7d37c7b","Type":"ContainerStarted","Data":"ef10d3b7fb7b882ba080e5ba4b5c2c255d1087eba1f18b5c55f188ee20f31ec5"} Sep 16 13:50:48.120998 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:48.120981 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7309-502317" podUID="8cabcf79-57f6-4e77-972e-df4585a6babd" containerName="test-curl-7309-502317" containerID="cri-o://22bb6b916813dec5892aad93d19c79204e2d27b7f4396f1bd61f550d8b4c6e7d" gracePeriod=2 Sep 16 13:50:48.121400 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:48.121053 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7309-9d65b0" event={"ID":"a599570b-1f08-4185-8e81-dcd3a7d37c7b","Type":"ContainerStarted","Data":"6809cdb3a2502f08ce24d2634647681c59072cde9be000f6f19b51cdd1b190c5"} Sep 16 13:50:48.144174 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:48.144132 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7309-9d65b0" podStartSLOduration=1.144117839 podStartE2EDuration="1.144117839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:50:48.141185225 +0000 UTC m=+1209.185488193" watchObservedRunningTime="2026-09-16 13:50:48.144117839 +0000 UTC m=+1209.188420804" Sep 16 13:50:48.282725 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:48.282700 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7309-ffee2f" Sep 16 13:50:48.424473 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:48.423958 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slq8d\" (UniqueName: \"kubernetes.io/projected/f3cd9ddc-afa6-4746-833c-338d7053546b-kube-api-access-slq8d\") pod \"f3cd9ddc-afa6-4746-833c-338d7053546b\" (UID: \"f3cd9ddc-afa6-4746-833c-338d7053546b\") " Sep 16 13:50:48.426938 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:48.426909 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3cd9ddc-afa6-4746-833c-338d7053546b-kube-api-access-slq8d" (OuterVolumeSpecName: "kube-api-access-slq8d") pod "f3cd9ddc-afa6-4746-833c-338d7053546b" (UID: "f3cd9ddc-afa6-4746-833c-338d7053546b"). InnerVolumeSpecName "kube-api-access-slq8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:50:48.525752 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:48.525722 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slq8d\" (UniqueName: \"kubernetes.io/projected/f3cd9ddc-afa6-4746-833c-338d7053546b-kube-api-access-slq8d\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:50:49.092339 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:49.092307 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7309-9d65b0"] Sep 16 13:50:49.098290 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:49.098218 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7309-9d65b0"] Sep 16 13:50:49.130250 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:49.130216 2562 generic.go:356] "Generic (PLEG): container finished" podID="f3cd9ddc-afa6-4746-833c-338d7053546b" containerID="bf5f7382f87c3a8f75d832f15f104cfc0b0fe51d6c31b7295953aa718757dc7f" exitCode=137 Sep 16 13:50:49.130691 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:49.130259 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7309-ffee2f" Sep 16 13:50:49.130691 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:49.130329 2562 scope.go:117] "RemoveContainer" containerID="bf5f7382f87c3a8f75d832f15f104cfc0b0fe51d6c31b7295953aa718757dc7f" Sep 16 13:50:49.130691 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:50:49.130579 2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7309-9d65b0" container="test-curl-7309-9d65b0" Sep 16 13:50:49.143033 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:50:49.143005 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7309-ffee2f\" is forbidden: User \"system:node:ip-10-0-139-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-23.ec2.internal' and this object" podUID="f3cd9ddc-afa6-4746-833c-338d7053546b" pod="opendatahub/test-curl-7309-ffee2f" Sep 16 13:50:49.143356 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:49.143341 2562 scope.go:117] "RemoveContainer" containerID="bf5f7382f87c3a8f75d832f15f104cfc0b0fe51d6c31b7295953aa718757dc7f" Sep 16 13:50:49.143763 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:50:49.143729 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf5f7382f87c3a8f75d832f15f104cfc0b0fe51d6c31b7295953aa718757dc7f\": container with ID starting with bf5f7382f87c3a8f75d832f15f104cfc0b0fe51d6c31b7295953aa718757dc7f not found: ID does not exist" containerID="bf5f7382f87c3a8f75d832f15f104cfc0b0fe51d6c31b7295953aa718757dc7f" Sep 16 13:50:49.143863 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:49.143762 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf5f7382f87c3a8f75d832f15f104cfc0b0fe51d6c31b7295953aa718757dc7f"} err="failed to get container status \"bf5f7382f87c3a8f75d832f15f104cfc0b0fe51d6c31b7295953aa718757dc7f\": rpc error: code = NotFound desc = could not find container \"bf5f7382f87c3a8f75d832f15f104cfc0b0fe51d6c31b7295953aa718757dc7f\": container with ID starting with bf5f7382f87c3a8f75d832f15f104cfc0b0fe51d6c31b7295953aa718757dc7f not found: ID does not exist" Sep 16 13:50:49.227505 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:49.227482 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7309-99afa9"] Sep 16 13:50:49.227941 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:49.227928 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a599570b-1f08-4185-8e81-dcd3a7d37c7b" containerName="test-curl-7309-9d65b0" Sep 16 13:50:49.227980 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:49.227943 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a599570b-1f08-4185-8e81-dcd3a7d37c7b" containerName="test-curl-7309-9d65b0" Sep 16 13:50:49.228022 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:49.228014 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a599570b-1f08-4185-8e81-dcd3a7d37c7b" containerName="test-curl-7309-9d65b0" Sep 16 13:50:49.232577 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:49.232562 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7309-99afa9" Sep 16 13:50:49.235938 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:50:49.235907 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7309-ffee2f\" is forbidden: User \"system:node:ip-10-0-139-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-23.ec2.internal' and this object" podUID="f3cd9ddc-afa6-4746-833c-338d7053546b" pod="opendatahub/test-curl-7309-ffee2f" Sep 16 13:50:49.243090 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:49.243067 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7309-99afa9"] Sep 16 13:50:49.331787 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:49.331760 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pksvw\" (UniqueName: \"kubernetes.io/projected/23fdb267-c865-4941-b79d-7eff7ea8304f-kube-api-access-pksvw\") pod \"test-curl-7309-99afa9\" (UID: \"23fdb267-c865-4941-b79d-7eff7ea8304f\") " pod="opendatahub/test-curl-7309-99afa9" Sep 16 13:50:49.413630 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:50:49.413592 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7309-ffee2f\" is forbidden: User \"system:node:ip-10-0-139-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-23.ec2.internal' and this object" podUID="f3cd9ddc-afa6-4746-833c-338d7053546b" pod="opendatahub/test-curl-7309-ffee2f" Sep 16 13:50:49.420700 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:49.420675 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3cd9ddc-afa6-4746-833c-338d7053546b" path="/var/lib/kubelet/pods/f3cd9ddc-afa6-4746-833c-338d7053546b/volumes" Sep 16 13:50:49.432872 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:49.432851 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pksvw\" (UniqueName: \"kubernetes.io/projected/23fdb267-c865-4941-b79d-7eff7ea8304f-kube-api-access-pksvw\") pod \"test-curl-7309-99afa9\" (UID: \"23fdb267-c865-4941-b79d-7eff7ea8304f\") " pod="opendatahub/test-curl-7309-99afa9" Sep 16 13:50:49.447338 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:49.447317 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pksvw\" (UniqueName: \"kubernetes.io/projected/23fdb267-c865-4941-b79d-7eff7ea8304f-kube-api-access-pksvw\") pod \"test-curl-7309-99afa9\" (UID: \"23fdb267-c865-4941-b79d-7eff7ea8304f\") " pod="opendatahub/test-curl-7309-99afa9" Sep 16 13:50:49.543134 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:49.543111 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7309-99afa9" Sep 16 13:50:49.830253 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:49.830227 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7309-99afa9"] Sep 16 13:50:49.831888 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:50:49.831863 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23fdb267_c865_4941_b79d_7eff7ea8304f.slice/crio-570d01b3455d923b58b6536c33af92f3011a770fa90c66e4f57049d6cd7eb74d WatchSource:0}: Error finding container 570d01b3455d923b58b6536c33af92f3011a770fa90c66e4f57049d6cd7eb74d: Status 404 returned error can't find the container with id 570d01b3455d923b58b6536c33af92f3011a770fa90c66e4f57049d6cd7eb74d Sep 16 13:50:50.139011 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:50.138985 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7309-99afa9" event={"ID":"23fdb267-c865-4941-b79d-7eff7ea8304f","Type":"ContainerStarted","Data":"58296efa0af92274d3d5fe6859789d6bfdada75034c11e6c17aa8d0e02a5e6d0"} Sep 16 13:50:50.139343 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:50.139023 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7309-99afa9" event={"ID":"23fdb267-c865-4941-b79d-7eff7ea8304f","Type":"ContainerStarted","Data":"570d01b3455d923b58b6536c33af92f3011a770fa90c66e4f57049d6cd7eb74d"} Sep 16 13:50:50.139343 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:50.139047 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7309-9d65b0" podUID="a599570b-1f08-4185-8e81-dcd3a7d37c7b" containerName="test-curl-7309-9d65b0" containerID="cri-o://ef10d3b7fb7b882ba080e5ba4b5c2c255d1087eba1f18b5c55f188ee20f31ec5" gracePeriod=2 Sep 16 13:50:50.164768 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:50.164729 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7309-99afa9" podStartSLOduration=1.164716551 podStartE2EDuration="1.164716551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:50:50.164408152 +0000 UTC m=+1211.208711115" watchObservedRunningTime="2026-09-16 13:50:50.164716551 +0000 UTC m=+1211.209019534" Sep 16 13:50:50.296526 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:50.296502 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7309-502317" Sep 16 13:50:50.442627 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:50.442597 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkx6s\" (UniqueName: \"kubernetes.io/projected/8cabcf79-57f6-4e77-972e-df4585a6babd-kube-api-access-dkx6s\") pod \"8cabcf79-57f6-4e77-972e-df4585a6babd\" (UID: \"8cabcf79-57f6-4e77-972e-df4585a6babd\") " Sep 16 13:50:50.444703 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:50.444680 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cabcf79-57f6-4e77-972e-df4585a6babd-kube-api-access-dkx6s" (OuterVolumeSpecName: "kube-api-access-dkx6s") pod "8cabcf79-57f6-4e77-972e-df4585a6babd" (UID: "8cabcf79-57f6-4e77-972e-df4585a6babd"). InnerVolumeSpecName "kube-api-access-dkx6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:50:50.543874 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:50.543720 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkx6s\" (UniqueName: \"kubernetes.io/projected/8cabcf79-57f6-4e77-972e-df4585a6babd-kube-api-access-dkx6s\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:50:50.855832 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:50.855757 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7309-99afa9"] Sep 16 13:50:50.861702 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:50.861676 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7309-99afa9"] Sep 16 13:50:51.108319 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.108246 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8"] Sep 16 13:50:51.108898 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.108882 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23fdb267-c865-4941-b79d-7eff7ea8304f" containerName="test-curl-7309-99afa9" Sep 16 13:50:51.108948 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.108902 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="23fdb267-c865-4941-b79d-7eff7ea8304f" containerName="test-curl-7309-99afa9" Sep 16 13:50:51.109074 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.109036 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="23fdb267-c865-4941-b79d-7eff7ea8304f" containerName="test-curl-7309-99afa9" Sep 16 13:50:51.112740 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.112719 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8" Sep 16 13:50:51.116981 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.116964 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-973610-serving-cert\"" Sep 16 13:50:51.123754 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.123728 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8"] Sep 16 13:50:51.147044 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.147016 2562 generic.go:356] "Generic (PLEG): container finished" podID="8cabcf79-57f6-4e77-972e-df4585a6babd" containerID="22bb6b916813dec5892aad93d19c79204e2d27b7f4396f1bd61f550d8b4c6e7d" exitCode=137 Sep 16 13:50:51.147425 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.147068 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7309-502317" Sep 16 13:50:51.147425 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.147088 2562 scope.go:117] "RemoveContainer" containerID="22bb6b916813dec5892aad93d19c79204e2d27b7f4396f1bd61f550d8b4c6e7d" Sep 16 13:50:51.147425 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:50:51.147358 2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7309-99afa9" container="test-curl-7309-99afa9" Sep 16 13:50:51.157533 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:50:51.157510 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7309-502317\" is forbidden: User \"system:node:ip-10-0-139-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-23.ec2.internal' and this object" podUID="8cabcf79-57f6-4e77-972e-df4585a6babd" pod="opendatahub/test-curl-7309-502317" Sep 16 13:50:51.158572 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.158557 2562 scope.go:117] "RemoveContainer" containerID="22bb6b916813dec5892aad93d19c79204e2d27b7f4396f1bd61f550d8b4c6e7d" Sep 16 13:50:51.158846 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:50:51.158828 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22bb6b916813dec5892aad93d19c79204e2d27b7f4396f1bd61f550d8b4c6e7d\": container with ID starting with 22bb6b916813dec5892aad93d19c79204e2d27b7f4396f1bd61f550d8b4c6e7d not found: ID does not exist" containerID="22bb6b916813dec5892aad93d19c79204e2d27b7f4396f1bd61f550d8b4c6e7d" Sep 16 13:50:51.158888 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.158854 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22bb6b916813dec5892aad93d19c79204e2d27b7f4396f1bd61f550d8b4c6e7d"} err="failed to get container status \"22bb6b916813dec5892aad93d19c79204e2d27b7f4396f1bd61f550d8b4c6e7d\": rpc error: code = NotFound desc = could not find container \"22bb6b916813dec5892aad93d19c79204e2d27b7f4396f1bd61f550d8b4c6e7d\": container with ID starting with 22bb6b916813dec5892aad93d19c79204e2d27b7f4396f1bd61f550d8b4c6e7d not found: ID does not exist" Sep 16 13:50:51.249048 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.249022 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1682ed-9061-4de4-8ebf-6c98237e1718-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8\" (UID: \"6f1682ed-9061-4de4-8ebf-6c98237e1718\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8" Sep 16 13:50:51.249209 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.249065 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn7xx\" (UniqueName: \"kubernetes.io/projected/6f1682ed-9061-4de4-8ebf-6c98237e1718-kube-api-access-nn7xx\") pod \"maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8\" (UID: \"6f1682ed-9061-4de4-8ebf-6c98237e1718\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8" Sep 16 13:50:51.249209 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.249138 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f1682ed-9061-4de4-8ebf-6c98237e1718-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8\" (UID: \"6f1682ed-9061-4de4-8ebf-6c98237e1718\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8" Sep 16 13:50:51.249209 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.249177 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f1682ed-9061-4de4-8ebf-6c98237e1718-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8\" (UID: \"6f1682ed-9061-4de4-8ebf-6c98237e1718\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8" Sep 16 13:50:51.350148 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.350121 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1682ed-9061-4de4-8ebf-6c98237e1718-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8\" (UID: \"6f1682ed-9061-4de4-8ebf-6c98237e1718\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8" Sep 16 13:50:51.350309 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.350170 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn7xx\" (UniqueName: \"kubernetes.io/projected/6f1682ed-9061-4de4-8ebf-6c98237e1718-kube-api-access-nn7xx\") pod \"maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8\" (UID: \"6f1682ed-9061-4de4-8ebf-6c98237e1718\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8" Sep 16 13:50:51.350309 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.350213 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f1682ed-9061-4de4-8ebf-6c98237e1718-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8\" (UID: \"6f1682ed-9061-4de4-8ebf-6c98237e1718\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8" Sep 16 13:50:51.350309 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.350239 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f1682ed-9061-4de4-8ebf-6c98237e1718-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8\" (UID: \"6f1682ed-9061-4de4-8ebf-6c98237e1718\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8" Sep 16 13:50:51.350851 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.350831 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1682ed-9061-4de4-8ebf-6c98237e1718-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8\" (UID: \"6f1682ed-9061-4de4-8ebf-6c98237e1718\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8" Sep 16 13:50:51.352808 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.352785 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f1682ed-9061-4de4-8ebf-6c98237e1718-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8\" (UID: \"6f1682ed-9061-4de4-8ebf-6c98237e1718\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8" Sep 16 13:50:51.352808 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.352805 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f1682ed-9061-4de4-8ebf-6c98237e1718-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8\" (UID: \"6f1682ed-9061-4de4-8ebf-6c98237e1718\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8" Sep 16 13:50:51.361542 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.361482 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn7xx\" (UniqueName: \"kubernetes.io/projected/6f1682ed-9061-4de4-8ebf-6c98237e1718-kube-api-access-nn7xx\") pod \"maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8\" (UID: \"6f1682ed-9061-4de4-8ebf-6c98237e1718\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8" Sep 16 13:50:51.414519 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.414495 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cabcf79-57f6-4e77-972e-df4585a6babd" path="/var/lib/kubelet/pods/8cabcf79-57f6-4e77-972e-df4585a6babd/volumes" Sep 16 13:50:51.424239 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.424219 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8" Sep 16 13:50:51.704553 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:51.704504 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8"] Sep 16 13:50:51.704697 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:50:51.704668 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f1682ed_9061_4de4_8ebf_6c98237e1718.slice/crio-0e0bed40e2b84e6f630a0c0970184d220f3e06d3f228aded12f4a14e4552d48f WatchSource:0}: Error finding container 0e0bed40e2b84e6f630a0c0970184d220f3e06d3f228aded12f4a14e4552d48f: Status 404 returned error can't find the container with id 0e0bed40e2b84e6f630a0c0970184d220f3e06d3f228aded12f4a14e4552d48f Sep 16 13:50:52.155974 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:52.155946 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8" event={"ID":"6f1682ed-9061-4de4-8ebf-6c98237e1718","Type":"ContainerStarted","Data":"0e0bed40e2b84e6f630a0c0970184d220f3e06d3f228aded12f4a14e4552d48f"} Sep 16 13:50:52.157198 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:52.157152 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7309-99afa9" podUID="23fdb267-c865-4941-b79d-7eff7ea8304f" containerName="test-curl-7309-99afa9" containerID="cri-o://58296efa0af92274d3d5fe6859789d6bfdada75034c11e6c17aa8d0e02a5e6d0" gracePeriod=2 Sep 16 13:50:52.305136 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:52.305110 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7309-9d65b0" Sep 16 13:50:52.461014 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:52.460992 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwvnk\" (UniqueName: \"kubernetes.io/projected/a599570b-1f08-4185-8e81-dcd3a7d37c7b-kube-api-access-lwvnk\") pod \"a599570b-1f08-4185-8e81-dcd3a7d37c7b\" (UID: \"a599570b-1f08-4185-8e81-dcd3a7d37c7b\") " Sep 16 13:50:52.463191 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:52.463168 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a599570b-1f08-4185-8e81-dcd3a7d37c7b-kube-api-access-lwvnk" (OuterVolumeSpecName: "kube-api-access-lwvnk") pod "a599570b-1f08-4185-8e81-dcd3a7d37c7b" (UID: "a599570b-1f08-4185-8e81-dcd3a7d37c7b"). InnerVolumeSpecName "kube-api-access-lwvnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:50:52.561840 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:52.561809 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwvnk\" (UniqueName: \"kubernetes.io/projected/a599570b-1f08-4185-8e81-dcd3a7d37c7b-kube-api-access-lwvnk\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:50:53.164041 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:53.164010 2562 generic.go:356] "Generic (PLEG): container finished" podID="a599570b-1f08-4185-8e81-dcd3a7d37c7b" containerID="ef10d3b7fb7b882ba080e5ba4b5c2c255d1087eba1f18b5c55f188ee20f31ec5" exitCode=137 Sep 16 13:50:53.164571 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:53.164059 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7309-9d65b0" Sep 16 13:50:53.164571 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:53.164109 2562 scope.go:117] "RemoveContainer" containerID="ef10d3b7fb7b882ba080e5ba4b5c2c255d1087eba1f18b5c55f188ee20f31ec5" Sep 16 13:50:53.166007 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:53.165982 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8" event={"ID":"6f1682ed-9061-4de4-8ebf-6c98237e1718","Type":"ContainerStarted","Data":"34830be5b2f44e124dfeb2e71794d545a82bfce0d244fd0fca9b1ed2b203fcdd"} Sep 16 13:50:53.166134 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:53.166037 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8" Sep 16 13:50:53.175951 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:53.175922 2562 scope.go:117] "RemoveContainer" containerID="ef10d3b7fb7b882ba080e5ba4b5c2c255d1087eba1f18b5c55f188ee20f31ec5" Sep 16 13:50:53.176270 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:50:53.176237 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef10d3b7fb7b882ba080e5ba4b5c2c255d1087eba1f18b5c55f188ee20f31ec5\": container with ID starting with ef10d3b7fb7b882ba080e5ba4b5c2c255d1087eba1f18b5c55f188ee20f31ec5 not found: ID does not exist" containerID="ef10d3b7fb7b882ba080e5ba4b5c2c255d1087eba1f18b5c55f188ee20f31ec5" Sep 16 13:50:53.176343 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:53.176282 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef10d3b7fb7b882ba080e5ba4b5c2c255d1087eba1f18b5c55f188ee20f31ec5"} err="failed to get container status \"ef10d3b7fb7b882ba080e5ba4b5c2c255d1087eba1f18b5c55f188ee20f31ec5\": rpc error: code = NotFound desc = could not find container \"ef10d3b7fb7b882ba080e5ba4b5c2c255d1087eba1f18b5c55f188ee20f31ec5\": container with ID starting with ef10d3b7fb7b882ba080e5ba4b5c2c255d1087eba1f18b5c55f188ee20f31ec5 not found: ID does not exist" Sep 16 13:50:53.188178 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:50:53.188151 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7309-9d65b0\" is forbidden: User \"system:node:ip-10-0-139-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-23.ec2.internal' and this object" podUID="a599570b-1f08-4185-8e81-dcd3a7d37c7b" pod="opendatahub/test-curl-7309-9d65b0" Sep 16 13:50:53.188340 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:53.188306 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8" podStartSLOduration=1.689756757 podStartE2EDuration="2.188293279s" totalImagesPullingTime="498.536522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:50:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:50:53.186013121 +0000 UTC m=+1214.230316107" watchObservedRunningTime="2026-09-16 13:50:53.188293279 +0000 UTC m=+1214.232596290" Sep 16 13:50:53.413703 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:53.413673 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a599570b-1f08-4185-8e81-dcd3a7d37c7b" path="/var/lib/kubelet/pods/a599570b-1f08-4185-8e81-dcd3a7d37c7b/volumes" Sep 16 13:50:54.336232 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:54.336207 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7309-99afa9" Sep 16 13:50:54.479859 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:54.479791 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pksvw\" (UniqueName: \"kubernetes.io/projected/23fdb267-c865-4941-b79d-7eff7ea8304f-kube-api-access-pksvw\") pod \"23fdb267-c865-4941-b79d-7eff7ea8304f\" (UID: \"23fdb267-c865-4941-b79d-7eff7ea8304f\") " Sep 16 13:50:54.481981 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:54.481959 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23fdb267-c865-4941-b79d-7eff7ea8304f-kube-api-access-pksvw" (OuterVolumeSpecName: "kube-api-access-pksvw") pod "23fdb267-c865-4941-b79d-7eff7ea8304f" (UID: "23fdb267-c865-4941-b79d-7eff7ea8304f"). InnerVolumeSpecName "kube-api-access-pksvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:50:54.580603 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:54.580580 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pksvw\" (UniqueName: \"kubernetes.io/projected/23fdb267-c865-4941-b79d-7eff7ea8304f-kube-api-access-pksvw\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:50:55.181480 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:55.181448 2562 generic.go:356] "Generic (PLEG): container finished" podID="23fdb267-c865-4941-b79d-7eff7ea8304f" containerID="58296efa0af92274d3d5fe6859789d6bfdada75034c11e6c17aa8d0e02a5e6d0" exitCode=137 Sep 16 13:50:55.181651 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:55.181490 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7309-99afa9" Sep 16 13:50:55.181651 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:55.181499 2562 scope.go:117] "RemoveContainer" containerID="58296efa0af92274d3d5fe6859789d6bfdada75034c11e6c17aa8d0e02a5e6d0" Sep 16 13:50:55.196122 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:50:55.196097 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7309-99afa9\" is forbidden: User \"system:node:ip-10-0-139-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-23.ec2.internal' and this object" podUID="23fdb267-c865-4941-b79d-7eff7ea8304f" pod="opendatahub/test-curl-7309-99afa9" Sep 16 13:50:55.197134 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:55.197118 2562 scope.go:117] "RemoveContainer" containerID="58296efa0af92274d3d5fe6859789d6bfdada75034c11e6c17aa8d0e02a5e6d0" Sep 16 13:50:55.197457 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:50:55.197437 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58296efa0af92274d3d5fe6859789d6bfdada75034c11e6c17aa8d0e02a5e6d0\": container with ID starting with 58296efa0af92274d3d5fe6859789d6bfdada75034c11e6c17aa8d0e02a5e6d0 not found: ID does not exist" containerID="58296efa0af92274d3d5fe6859789d6bfdada75034c11e6c17aa8d0e02a5e6d0" Sep 16 13:50:55.197539 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:55.197474 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58296efa0af92274d3d5fe6859789d6bfdada75034c11e6c17aa8d0e02a5e6d0"} err="failed to get container status \"58296efa0af92274d3d5fe6859789d6bfdada75034c11e6c17aa8d0e02a5e6d0\": rpc error: code = NotFound desc = could not find container \"58296efa0af92274d3d5fe6859789d6bfdada75034c11e6c17aa8d0e02a5e6d0\": container with ID starting with 58296efa0af92274d3d5fe6859789d6bfdada75034c11e6c17aa8d0e02a5e6d0 not found: ID does not exist" Sep 16 13:50:55.414305 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:55.414275 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23fdb267-c865-4941-b79d-7eff7ea8304f" path="/var/lib/kubelet/pods/23fdb267-c865-4941-b79d-7eff7ea8304f/volumes" Sep 16 13:50:59.179868 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:50:59.179842 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8" Sep 16 13:51:02.434577 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:02.434547 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f"] Sep 16 13:51:02.439210 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:02.439184 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f" Sep 16 13:51:02.442814 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:02.442793 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c5dc2482-serving-cert\"" Sep 16 13:51:02.463966 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:02.463943 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f"] Sep 16 13:51:02.557686 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:02.557664 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc515c2d-9c24-4677-bc3b-984619df3e43-maas-api-tls\") pod \"maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f\" (UID: \"dc515c2d-9c24-4677-bc3b-984619df3e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f" Sep 16 13:51:02.557844 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:02.557703 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc515c2d-9c24-4677-bc3b-984619df3e43-openshift-service-ca\") pod \"maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f\" (UID: \"dc515c2d-9c24-4677-bc3b-984619df3e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f" Sep 16 13:51:02.557844 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:02.557777 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc515c2d-9c24-4677-bc3b-984619df3e43-metrics-certs\") pod \"maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f\" (UID: \"dc515c2d-9c24-4677-bc3b-984619df3e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f" Sep 16 13:51:02.557940 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:02.557843 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqhc9\" (UniqueName: \"kubernetes.io/projected/dc515c2d-9c24-4677-bc3b-984619df3e43-kube-api-access-tqhc9\") pod \"maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f\" (UID: \"dc515c2d-9c24-4677-bc3b-984619df3e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f" Sep 16 13:51:02.658854 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:02.658823 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc515c2d-9c24-4677-bc3b-984619df3e43-maas-api-tls\") pod \"maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f\" (UID: \"dc515c2d-9c24-4677-bc3b-984619df3e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f" Sep 16 13:51:02.659033 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:02.658871 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc515c2d-9c24-4677-bc3b-984619df3e43-openshift-service-ca\") pod \"maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f\" (UID: \"dc515c2d-9c24-4677-bc3b-984619df3e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f" Sep 16 13:51:02.659033 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:02.658914 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc515c2d-9c24-4677-bc3b-984619df3e43-metrics-certs\") pod \"maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f\" (UID: \"dc515c2d-9c24-4677-bc3b-984619df3e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f" Sep 16 13:51:02.659033 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:02.658938 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqhc9\" (UniqueName: \"kubernetes.io/projected/dc515c2d-9c24-4677-bc3b-984619df3e43-kube-api-access-tqhc9\") pod \"maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f\" (UID: \"dc515c2d-9c24-4677-bc3b-984619df3e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f" Sep 16 13:51:02.659531 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:02.659511 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc515c2d-9c24-4677-bc3b-984619df3e43-openshift-service-ca\") pod \"maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f\" (UID: \"dc515c2d-9c24-4677-bc3b-984619df3e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f" Sep 16 13:51:02.661466 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:02.661441 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc515c2d-9c24-4677-bc3b-984619df3e43-metrics-certs\") pod \"maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f\" (UID: \"dc515c2d-9c24-4677-bc3b-984619df3e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f" Sep 16 13:51:02.661559 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:02.661471 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc515c2d-9c24-4677-bc3b-984619df3e43-maas-api-tls\") pod \"maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f\" (UID: \"dc515c2d-9c24-4677-bc3b-984619df3e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f" Sep 16 13:51:02.671237 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:02.671215 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqhc9\" (UniqueName: \"kubernetes.io/projected/dc515c2d-9c24-4677-bc3b-984619df3e43-kube-api-access-tqhc9\") pod \"maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f\" (UID: \"dc515c2d-9c24-4677-bc3b-984619df3e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f" Sep 16 13:51:02.750054 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:02.750000 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f" Sep 16 13:51:03.030346 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:03.030315 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f"] Sep 16 13:51:03.031413 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:51:03.031366 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc515c2d_9c24_4677_bc3b_984619df3e43.slice/crio-0b18cc11b189ada10d024465efc9211ac19a69309fdb1c21c3b80a506b1f67f2 WatchSource:0}: Error finding container 0b18cc11b189ada10d024465efc9211ac19a69309fdb1c21c3b80a506b1f67f2: Status 404 returned error can't find the container with id 0b18cc11b189ada10d024465efc9211ac19a69309fdb1c21c3b80a506b1f67f2 Sep 16 13:51:03.230316 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:03.230282 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f" event={"ID":"dc515c2d-9c24-4677-bc3b-984619df3e43","Type":"ContainerStarted","Data":"0b18cc11b189ada10d024465efc9211ac19a69309fdb1c21c3b80a506b1f67f2"} Sep 16 13:51:04.237935 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:04.237899 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f" event={"ID":"dc515c2d-9c24-4677-bc3b-984619df3e43","Type":"ContainerStarted","Data":"6ad445a66385a29c3521d96f6808dda74dabd0fdd429e6f6abe9f0a8ae7963ba"} Sep 16 13:51:04.238328 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:04.237977 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f" Sep 16 13:51:04.267323 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:04.267283 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f" podStartSLOduration=1.8453185890000001 podStartE2EDuration="2.267268829s" totalImagesPullingTime="421.95024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:51:04.266206116 +0000 UTC m=+1225.310509093" watchObservedRunningTime="2026-09-16 13:51:04.267268829 +0000 UTC m=+1225.311571794" Sep 16 13:51:10.250772 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:10.250742 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f" Sep 16 13:51:22.234918 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:22.234888 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5"] Sep 16 13:51:22.241979 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:22.241961 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" Sep 16 13:51:22.245113 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:22.245092 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-8c36a6-serving-cert\"" Sep 16 13:51:22.260541 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:22.260520 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5"] Sep 16 13:51:22.342285 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:22.342254 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04269b40-58d6-47d2-b577-96a84c90f3bf-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5\" (UID: \"04269b40-58d6-47d2-b577-96a84c90f3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" Sep 16 13:51:22.342512 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:22.342329 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hzl9\" (UniqueName: \"kubernetes.io/projected/04269b40-58d6-47d2-b577-96a84c90f3bf-kube-api-access-4hzl9\") pod \"maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5\" (UID: \"04269b40-58d6-47d2-b577-96a84c90f3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" Sep 16 13:51:22.342512 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:22.342400 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04269b40-58d6-47d2-b577-96a84c90f3bf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5\" (UID: \"04269b40-58d6-47d2-b577-96a84c90f3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" Sep 16 13:51:22.342512 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:22.342463 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04269b40-58d6-47d2-b577-96a84c90f3bf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5\" (UID: \"04269b40-58d6-47d2-b577-96a84c90f3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" Sep 16 13:51:22.443943 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:22.443912 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04269b40-58d6-47d2-b577-96a84c90f3bf-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5\" (UID: \"04269b40-58d6-47d2-b577-96a84c90f3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" Sep 16 13:51:22.444082 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:22.443954 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hzl9\" (UniqueName: \"kubernetes.io/projected/04269b40-58d6-47d2-b577-96a84c90f3bf-kube-api-access-4hzl9\") pod \"maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5\" (UID: \"04269b40-58d6-47d2-b577-96a84c90f3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" Sep 16 13:51:22.444082 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:22.443984 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04269b40-58d6-47d2-b577-96a84c90f3bf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5\" (UID: \"04269b40-58d6-47d2-b577-96a84c90f3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" Sep 16 13:51:22.444082 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:22.444018 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04269b40-58d6-47d2-b577-96a84c90f3bf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5\" (UID: \"04269b40-58d6-47d2-b577-96a84c90f3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" Sep 16 13:51:22.444188 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:51:22.444125 2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-serving-cert: secret "maas-api-e2e-shared-b-w3-8c36a6-serving-cert" not found Sep 16 13:51:22.444188 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:51:22.444179 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04269b40-58d6-47d2-b577-96a84c90f3bf-maas-api-tls podName:04269b40-58d6-47d2-b577-96a84c90f3bf nodeName:}" failed. No retries permitted until 2026-09-16 13:51:22.944162209 +0000 UTC m=+1243.988465155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/04269b40-58d6-47d2-b577-96a84c90f3bf-maas-api-tls") pod "maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" (UID: "04269b40-58d6-47d2-b577-96a84c90f3bf") : secret "maas-api-e2e-shared-b-w3-8c36a6-serving-cert" not found Sep 16 13:51:22.444676 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:22.444650 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04269b40-58d6-47d2-b577-96a84c90f3bf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5\" (UID: \"04269b40-58d6-47d2-b577-96a84c90f3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" Sep 16 13:51:22.446458 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:22.446441 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04269b40-58d6-47d2-b577-96a84c90f3bf-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5\" (UID: \"04269b40-58d6-47d2-b577-96a84c90f3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" Sep 16 13:51:22.455287 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:22.455261 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hzl9\" (UniqueName: \"kubernetes.io/projected/04269b40-58d6-47d2-b577-96a84c90f3bf-kube-api-access-4hzl9\") pod \"maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5\" (UID: \"04269b40-58d6-47d2-b577-96a84c90f3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" Sep 16 13:51:22.949632 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:22.949594 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04269b40-58d6-47d2-b577-96a84c90f3bf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5\" (UID: \"04269b40-58d6-47d2-b577-96a84c90f3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" Sep 16 13:51:22.952121 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:22.952097 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04269b40-58d6-47d2-b577-96a84c90f3bf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5\" (UID: \"04269b40-58d6-47d2-b577-96a84c90f3bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" Sep 16 13:51:23.152679 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:23.152644 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" Sep 16 13:51:23.428691 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:23.428645 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5"] Sep 16 13:51:23.429197 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:51:23.429175 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04269b40_58d6_47d2_b577_96a84c90f3bf.slice/crio-045d2f9b192013025c3f64ad6ba769f0b26281e19212a1a9675579e3c3c00405 WatchSource:0}: Error finding container 045d2f9b192013025c3f64ad6ba769f0b26281e19212a1a9675579e3c3c00405: Status 404 returned error can't find the container with id 045d2f9b192013025c3f64ad6ba769f0b26281e19212a1a9675579e3c3c00405 Sep 16 13:51:24.361303 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:24.361261 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" event={"ID":"04269b40-58d6-47d2-b577-96a84c90f3bf","Type":"ContainerStarted","Data":"8e8a8fc9fd0c6033035e068226f04a3daf4d2c8239ffb0a5d0e1639f13e2bc74"} Sep 16 13:51:24.361496 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:24.361461 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" Sep 16 13:51:24.361547 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:24.361498 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" event={"ID":"04269b40-58d6-47d2-b577-96a84c90f3bf","Type":"ContainerStarted","Data":"045d2f9b192013025c3f64ad6ba769f0b26281e19212a1a9675579e3c3c00405"} Sep 16 13:51:24.386036 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:24.385986 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" podStartSLOduration=2.002469638 podStartE2EDuration="2.385972867s" totalImagesPullingTime="383.503229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:51:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:51:24.38219578 +0000 UTC m=+1245.426498762" watchObservedRunningTime="2026-09-16 13:51:24.385972867 +0000 UTC m=+1245.430275860" Sep 16 13:51:30.374945 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:51:30.374908 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" Sep 16 13:52:47.379987 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:47.379958 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f"] Sep 16 13:52:47.380602 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:47.380192 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f" podUID="dc515c2d-9c24-4677-bc3b-984619df3e43" containerName="maas-api" containerID="cri-o://6ad445a66385a29c3521d96f6808dda74dabd0fdd429e6f6abe9f0a8ae7963ba" gracePeriod=30 Sep 16 13:52:47.619135 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:47.619110 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f" Sep 16 13:52:47.781067 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:47.781029 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqhc9\" (UniqueName: \"kubernetes.io/projected/dc515c2d-9c24-4677-bc3b-984619df3e43-kube-api-access-tqhc9\") pod \"dc515c2d-9c24-4677-bc3b-984619df3e43\" (UID: \"dc515c2d-9c24-4677-bc3b-984619df3e43\") " Sep 16 13:52:47.781263 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:47.781102 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc515c2d-9c24-4677-bc3b-984619df3e43-maas-api-tls\") pod \"dc515c2d-9c24-4677-bc3b-984619df3e43\" (UID: \"dc515c2d-9c24-4677-bc3b-984619df3e43\") " Sep 16 13:52:47.781263 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:47.781167 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc515c2d-9c24-4677-bc3b-984619df3e43-openshift-service-ca\") pod \"dc515c2d-9c24-4677-bc3b-984619df3e43\" (UID: \"dc515c2d-9c24-4677-bc3b-984619df3e43\") " Sep 16 13:52:47.781432 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:47.781278 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc515c2d-9c24-4677-bc3b-984619df3e43-metrics-certs\") pod \"dc515c2d-9c24-4677-bc3b-984619df3e43\" (UID: \"dc515c2d-9c24-4677-bc3b-984619df3e43\") " Sep 16 13:52:47.781796 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:47.781766 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc515c2d-9c24-4677-bc3b-984619df3e43-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dc515c2d-9c24-4677-bc3b-984619df3e43" (UID: "dc515c2d-9c24-4677-bc3b-984619df3e43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:52:47.781955 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:47.781935 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc515c2d-9c24-4677-bc3b-984619df3e43-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:52:47.783796 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:47.783771 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc515c2d-9c24-4677-bc3b-984619df3e43-kube-api-access-tqhc9" (OuterVolumeSpecName: "kube-api-access-tqhc9") pod "dc515c2d-9c24-4677-bc3b-984619df3e43" (UID: "dc515c2d-9c24-4677-bc3b-984619df3e43"). InnerVolumeSpecName "kube-api-access-tqhc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:52:47.784094 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:47.784066 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc515c2d-9c24-4677-bc3b-984619df3e43-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dc515c2d-9c24-4677-bc3b-984619df3e43" (UID: "dc515c2d-9c24-4677-bc3b-984619df3e43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:52:47.784295 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:47.784214 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc515c2d-9c24-4677-bc3b-984619df3e43-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dc515c2d-9c24-4677-bc3b-984619df3e43" (UID: "dc515c2d-9c24-4677-bc3b-984619df3e43"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:52:47.857925 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:47.857895 2562 generic.go:356] "Generic (PLEG): container finished" podID="dc515c2d-9c24-4677-bc3b-984619df3e43" containerID="6ad445a66385a29c3521d96f6808dda74dabd0fdd429e6f6abe9f0a8ae7963ba" exitCode=0 Sep 16 13:52:47.858051 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:47.857940 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f" Sep 16 13:52:47.858051 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:47.857962 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f" event={"ID":"dc515c2d-9c24-4677-bc3b-984619df3e43","Type":"ContainerDied","Data":"6ad445a66385a29c3521d96f6808dda74dabd0fdd429e6f6abe9f0a8ae7963ba"} Sep 16 13:52:47.858051 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:47.858004 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f" event={"ID":"dc515c2d-9c24-4677-bc3b-984619df3e43","Type":"ContainerDied","Data":"0b18cc11b189ada10d024465efc9211ac19a69309fdb1c21c3b80a506b1f67f2"} Sep 16 13:52:47.858051 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:47.858025 2562 scope.go:117] "RemoveContainer" containerID="6ad445a66385a29c3521d96f6808dda74dabd0fdd429e6f6abe9f0a8ae7963ba" Sep 16 13:52:47.868015 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:47.867995 2562 scope.go:117] "RemoveContainer" containerID="6ad445a66385a29c3521d96f6808dda74dabd0fdd429e6f6abe9f0a8ae7963ba" Sep 16 13:52:47.868283 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:52:47.868265 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ad445a66385a29c3521d96f6808dda74dabd0fdd429e6f6abe9f0a8ae7963ba\": container with ID starting with 6ad445a66385a29c3521d96f6808dda74dabd0fdd429e6f6abe9f0a8ae7963ba not found: ID does not exist" containerID="6ad445a66385a29c3521d96f6808dda74dabd0fdd429e6f6abe9f0a8ae7963ba" Sep 16 13:52:47.868328 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:47.868291 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad445a66385a29c3521d96f6808dda74dabd0fdd429e6f6abe9f0a8ae7963ba"} err="failed to get container status \"6ad445a66385a29c3521d96f6808dda74dabd0fdd429e6f6abe9f0a8ae7963ba\": rpc error: code = NotFound desc = could not find container \"6ad445a66385a29c3521d96f6808dda74dabd0fdd429e6f6abe9f0a8ae7963ba\": container with ID starting with 6ad445a66385a29c3521d96f6808dda74dabd0fdd429e6f6abe9f0a8ae7963ba not found: ID does not exist" Sep 16 13:52:47.880246 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:47.880224 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f"] Sep 16 13:52:47.882971 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:47.882951 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqhc9\" (UniqueName: \"kubernetes.io/projected/dc515c2d-9c24-4677-bc3b-984619df3e43-kube-api-access-tqhc9\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:52:47.882971 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:47.882972 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc515c2d-9c24-4677-bc3b-984619df3e43-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:52:47.883092 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:47.882980 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc515c2d-9c24-4677-bc3b-984619df3e43-metrics-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:52:47.884386 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:47.884358 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c5dc2482-7679484cc-rzd7f"] Sep 16 13:52:49.414205 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:52:49.414164 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc515c2d-9c24-4677-bc3b-984619df3e43" path="/var/lib/kubelet/pods/dc515c2d-9c24-4677-bc3b-984619df3e43/volumes" Sep 16 13:54:13.084461 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.084428 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6"] Sep 16 13:54:13.084913 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.084856 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc515c2d-9c24-4677-bc3b-984619df3e43" containerName="maas-api" Sep 16 13:54:13.084913 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.084868 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc515c2d-9c24-4677-bc3b-984619df3e43" containerName="maas-api" Sep 16 13:54:13.084982 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.084951 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc515c2d-9c24-4677-bc3b-984619df3e43" containerName="maas-api" Sep 16 13:54:13.088122 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.088104 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:54:13.090978 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.090957 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-46ccbd\"/\"embedding-w0-46ccbd-kserve-self-signed-certs\"" Sep 16 13:54:13.098580 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.098561 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6"] Sep 16 13:54:13.163441 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.163409 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df453e94-cc90-4e00-a3f0-7106c99da452-home\") pod \"embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6\" (UID: \"df453e94-cc90-4e00-a3f0-7106c99da452\") " pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:54:13.163595 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.163467 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df453e94-cc90-4e00-a3f0-7106c99da452-model-cache\") pod \"embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6\" (UID: \"df453e94-cc90-4e00-a3f0-7106c99da452\") " pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:54:13.163595 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.163534 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df453e94-cc90-4e00-a3f0-7106c99da452-dshm\") pod \"embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6\" (UID: \"df453e94-cc90-4e00-a3f0-7106c99da452\") " pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:54:13.163688 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.163650 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkf6j\" (UniqueName: \"kubernetes.io/projected/df453e94-cc90-4e00-a3f0-7106c99da452-kube-api-access-vkf6j\") pod \"embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6\" (UID: \"df453e94-cc90-4e00-a3f0-7106c99da452\") " pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:54:13.163732 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.163696 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df453e94-cc90-4e00-a3f0-7106c99da452-tls-certs\") pod \"embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6\" (UID: \"df453e94-cc90-4e00-a3f0-7106c99da452\") " pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:54:13.163732 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.163723 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df453e94-cc90-4e00-a3f0-7106c99da452-tmp-dir\") pod \"embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6\" (UID: \"df453e94-cc90-4e00-a3f0-7106c99da452\") " pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:54:13.264613 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.264571 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df453e94-cc90-4e00-a3f0-7106c99da452-dshm\") pod \"embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6\" (UID: \"df453e94-cc90-4e00-a3f0-7106c99da452\") " pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:54:13.264814 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.264684 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkf6j\" (UniqueName: \"kubernetes.io/projected/df453e94-cc90-4e00-a3f0-7106c99da452-kube-api-access-vkf6j\") pod \"embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6\" (UID: \"df453e94-cc90-4e00-a3f0-7106c99da452\") " pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:54:13.264814 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.264721 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df453e94-cc90-4e00-a3f0-7106c99da452-tls-certs\") pod \"embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6\" (UID: \"df453e94-cc90-4e00-a3f0-7106c99da452\") " pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:54:13.264947 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.264850 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df453e94-cc90-4e00-a3f0-7106c99da452-tmp-dir\") pod \"embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6\" (UID: \"df453e94-cc90-4e00-a3f0-7106c99da452\") " pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:54:13.264947 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.264913 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df453e94-cc90-4e00-a3f0-7106c99da452-home\") pod \"embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6\" (UID: \"df453e94-cc90-4e00-a3f0-7106c99da452\") " pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:54:13.265053 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.264960 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df453e94-cc90-4e00-a3f0-7106c99da452-model-cache\") pod \"embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6\" (UID: \"df453e94-cc90-4e00-a3f0-7106c99da452\") " pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:54:13.265185 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.265168 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df453e94-cc90-4e00-a3f0-7106c99da452-tmp-dir\") pod \"embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6\" (UID: \"df453e94-cc90-4e00-a3f0-7106c99da452\") " pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:54:13.265360 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.265297 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df453e94-cc90-4e00-a3f0-7106c99da452-home\") pod \"embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6\" (UID: \"df453e94-cc90-4e00-a3f0-7106c99da452\") " pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:54:13.265502 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.265337 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df453e94-cc90-4e00-a3f0-7106c99da452-model-cache\") pod \"embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6\" (UID: \"df453e94-cc90-4e00-a3f0-7106c99da452\") " pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:54:13.267027 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.267006 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df453e94-cc90-4e00-a3f0-7106c99da452-dshm\") pod \"embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6\" (UID: \"df453e94-cc90-4e00-a3f0-7106c99da452\") " pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:54:13.267296 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.267280 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df453e94-cc90-4e00-a3f0-7106c99da452-tls-certs\") pod \"embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6\" (UID: \"df453e94-cc90-4e00-a3f0-7106c99da452\") " pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:54:13.273617 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.273601 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkf6j\" (UniqueName: \"kubernetes.io/projected/df453e94-cc90-4e00-a3f0-7106c99da452-kube-api-access-vkf6j\") pod \"embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6\" (UID: \"df453e94-cc90-4e00-a3f0-7106c99da452\") " pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:54:13.398955 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.398885 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:54:13.670853 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:13.670825 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6"] Sep 16 13:54:13.672052 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:54:13.672030 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf453e94_cc90_4e00_a3f0_7106c99da452.slice/crio-697e61f0b5baec85802d2a554fb37fc80e1de0bf8c6e3386fc54eb94d1905a71 WatchSource:0}: Error finding container 697e61f0b5baec85802d2a554fb37fc80e1de0bf8c6e3386fc54eb94d1905a71: Status 404 returned error can't find the container with id 697e61f0b5baec85802d2a554fb37fc80e1de0bf8c6e3386fc54eb94d1905a71 Sep 16 13:54:14.368177 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:14.368128 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" event={"ID":"df453e94-cc90-4e00-a3f0-7106c99da452","Type":"ContainerStarted","Data":"e1178ae8e904ac6a6e9b2d77853bfd7f64a0740c54fc618ba910a969d68d0398"} Sep 16 13:54:14.368177 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:14.368181 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" event={"ID":"df453e94-cc90-4e00-a3f0-7106c99da452","Type":"ContainerStarted","Data":"697e61f0b5baec85802d2a554fb37fc80e1de0bf8c6e3386fc54eb94d1905a71"} Sep 16 13:54:14.385358 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:14.385263 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" podStartSLOduration=1.385245853 podStartE2EDuration="1.385245853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:54:14.38456855 +0000 UTC m=+1415.428871517" watchObservedRunningTime="2026-09-16 13:54:14.385245853 +0000 UTC m=+1415.429548819" Sep 16 13:54:23.399744 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:23.399708 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:54:23.402260 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:23.399759 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:54:23.413861 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:23.413841 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:54:23.435895 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:54:23.435875 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:55:24.406188 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:24.406151 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr"] Sep 16 13:55:24.410208 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:24.410188 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" Sep 16 13:55:24.412510 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:24.412490 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-bdd04e-serving-cert\"" Sep 16 13:55:24.419618 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:24.419596 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr"] Sep 16 13:55:24.419813 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:24.419797 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5427392c-91e4-45ea-bc64-9d9ca1f5b404-openshift-service-ca\") pod \"maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr\" (UID: \"5427392c-91e4-45ea-bc64-9d9ca1f5b404\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" Sep 16 13:55:24.419863 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:24.419829 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5427392c-91e4-45ea-bc64-9d9ca1f5b404-maas-api-tls\") pod \"maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr\" (UID: \"5427392c-91e4-45ea-bc64-9d9ca1f5b404\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" Sep 16 13:55:24.419901 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:24.419889 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5427392c-91e4-45ea-bc64-9d9ca1f5b404-metrics-certs\") pod \"maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr\" (UID: \"5427392c-91e4-45ea-bc64-9d9ca1f5b404\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" Sep 16 13:55:24.420115 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:24.420054 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r7mt\" (UniqueName: \"kubernetes.io/projected/5427392c-91e4-45ea-bc64-9d9ca1f5b404-kube-api-access-6r7mt\") pod \"maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr\" (UID: \"5427392c-91e4-45ea-bc64-9d9ca1f5b404\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" Sep 16 13:55:24.520771 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:24.520740 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5427392c-91e4-45ea-bc64-9d9ca1f5b404-openshift-service-ca\") pod \"maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr\" (UID: \"5427392c-91e4-45ea-bc64-9d9ca1f5b404\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" Sep 16 13:55:24.520771 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:24.520776 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5427392c-91e4-45ea-bc64-9d9ca1f5b404-maas-api-tls\") pod \"maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr\" (UID: \"5427392c-91e4-45ea-bc64-9d9ca1f5b404\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" Sep 16 13:55:24.520998 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:24.520801 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5427392c-91e4-45ea-bc64-9d9ca1f5b404-metrics-certs\") pod \"maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr\" (UID: \"5427392c-91e4-45ea-bc64-9d9ca1f5b404\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" Sep 16 13:55:24.520998 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:24.520870 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r7mt\" (UniqueName: \"kubernetes.io/projected/5427392c-91e4-45ea-bc64-9d9ca1f5b404-kube-api-access-6r7mt\") pod \"maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr\" (UID: \"5427392c-91e4-45ea-bc64-9d9ca1f5b404\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" Sep 16 13:55:24.520998 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:55:24.520892 2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-serving-cert: secret "maas-api-e2e-api-a-bdd04e-serving-cert" not found Sep 16 13:55:24.520998 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:55:24.520960 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5427392c-91e4-45ea-bc64-9d9ca1f5b404-maas-api-tls podName:5427392c-91e4-45ea-bc64-9d9ca1f5b404 nodeName:}" failed. No retries permitted until 2026-09-16 13:55:25.020944325 +0000 UTC m=+1486.065247294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5427392c-91e4-45ea-bc64-9d9ca1f5b404-maas-api-tls") pod "maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" (UID: "5427392c-91e4-45ea-bc64-9d9ca1f5b404") : secret "maas-api-e2e-api-a-bdd04e-serving-cert" not found Sep 16 13:55:24.521488 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:24.521464 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5427392c-91e4-45ea-bc64-9d9ca1f5b404-openshift-service-ca\") pod \"maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr\" (UID: \"5427392c-91e4-45ea-bc64-9d9ca1f5b404\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" Sep 16 13:55:24.523395 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:24.523362 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5427392c-91e4-45ea-bc64-9d9ca1f5b404-metrics-certs\") pod \"maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr\" (UID: \"5427392c-91e4-45ea-bc64-9d9ca1f5b404\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" Sep 16 13:55:24.531634 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:24.531614 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r7mt\" (UniqueName: \"kubernetes.io/projected/5427392c-91e4-45ea-bc64-9d9ca1f5b404-kube-api-access-6r7mt\") pod \"maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr\" (UID: \"5427392c-91e4-45ea-bc64-9d9ca1f5b404\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" Sep 16 13:55:25.025587 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:25.025557 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5427392c-91e4-45ea-bc64-9d9ca1f5b404-maas-api-tls\") pod \"maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr\" (UID: \"5427392c-91e4-45ea-bc64-9d9ca1f5b404\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" Sep 16 13:55:25.028039 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:25.028017 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5427392c-91e4-45ea-bc64-9d9ca1f5b404-maas-api-tls\") pod \"maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr\" (UID: \"5427392c-91e4-45ea-bc64-9d9ca1f5b404\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" Sep 16 13:55:25.322330 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:25.322243 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" Sep 16 13:55:25.599008 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:25.598983 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr"] Sep 16 13:55:25.600792 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:55:25.600762 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5427392c_91e4_45ea_bc64_9d9ca1f5b404.slice/crio-ff76771f20d8b23a0df12de376c3a97d9e83198441c02f840a9a968934378f05 WatchSource:0}: Error finding container ff76771f20d8b23a0df12de376c3a97d9e83198441c02f840a9a968934378f05: Status 404 returned error can't find the container with id ff76771f20d8b23a0df12de376c3a97d9e83198441c02f840a9a968934378f05 Sep 16 13:55:25.602227 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:25.602210 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 13:55:25.793443 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:25.793408 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" event={"ID":"5427392c-91e4-45ea-bc64-9d9ca1f5b404","Type":"ContainerStarted","Data":"ff76771f20d8b23a0df12de376c3a97d9e83198441c02f840a9a968934378f05"} Sep 16 13:55:26.801025 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:26.800982 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" event={"ID":"5427392c-91e4-45ea-bc64-9d9ca1f5b404","Type":"ContainerStarted","Data":"635d3d3fd43f5b6862cf2321ca436c9970442f8fdfa2fd22fd62d00f72034102"} Sep 16 13:55:26.801505 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:26.801051 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" Sep 16 13:55:26.823176 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:26.823127 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" podStartSLOduration=2.344722205 podStartE2EDuration="2.823111293s" totalImagesPullingTime="478.389088ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:55:26.820500926 +0000 UTC m=+1487.864803915" watchObservedRunningTime="2026-09-16 13:55:26.823111293 +0000 UTC m=+1487.867414259" Sep 16 13:55:32.812291 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:32.812265 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" Sep 16 13:55:39.529575 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:39.529550 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4xmxv_c7828efb-6be7-4f7b-986f-645a2370163a/console-operator/1.log" Sep 16 13:55:39.535429 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:39.535407 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4xmxv_c7828efb-6be7-4f7b-986f-645a2370163a/console-operator/1.log" Sep 16 13:55:59.608621 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:59.608582 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd"] Sep 16 13:55:59.614431 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:59.614403 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" Sep 16 13:55:59.617209 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:59.617185 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-59a5a8-serving-cert\"" Sep 16 13:55:59.620347 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:59.620325 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd"] Sep 16 13:55:59.667673 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:59.667643 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22fa4c0-cdee-42dd-ba5d-52ece500282c-metrics-certs\") pod \"maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd\" (UID: \"a22fa4c0-cdee-42dd-ba5d-52ece500282c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" Sep 16 13:55:59.667834 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:59.667742 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a22fa4c0-cdee-42dd-ba5d-52ece500282c-maas-api-tls\") pod \"maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd\" (UID: \"a22fa4c0-cdee-42dd-ba5d-52ece500282c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" Sep 16 13:55:59.667834 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:59.667800 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf85p\" (UniqueName: \"kubernetes.io/projected/a22fa4c0-cdee-42dd-ba5d-52ece500282c-kube-api-access-sf85p\") pod \"maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd\" (UID: \"a22fa4c0-cdee-42dd-ba5d-52ece500282c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" Sep 16 13:55:59.667834 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:59.667830 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22fa4c0-cdee-42dd-ba5d-52ece500282c-openshift-service-ca\") pod \"maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd\" (UID: \"a22fa4c0-cdee-42dd-ba5d-52ece500282c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" Sep 16 13:55:59.768835 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:59.768802 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf85p\" (UniqueName: \"kubernetes.io/projected/a22fa4c0-cdee-42dd-ba5d-52ece500282c-kube-api-access-sf85p\") pod \"maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd\" (UID: \"a22fa4c0-cdee-42dd-ba5d-52ece500282c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" Sep 16 13:55:59.769026 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:59.768865 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22fa4c0-cdee-42dd-ba5d-52ece500282c-openshift-service-ca\") pod \"maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd\" (UID: \"a22fa4c0-cdee-42dd-ba5d-52ece500282c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" Sep 16 13:55:59.769026 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:59.768928 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22fa4c0-cdee-42dd-ba5d-52ece500282c-metrics-certs\") pod \"maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd\" (UID: \"a22fa4c0-cdee-42dd-ba5d-52ece500282c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" Sep 16 13:55:59.769026 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:59.769017 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a22fa4c0-cdee-42dd-ba5d-52ece500282c-maas-api-tls\") pod \"maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd\" (UID: \"a22fa4c0-cdee-42dd-ba5d-52ece500282c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" Sep 16 13:55:59.769180 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:55:59.769110 2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-serving-cert: secret "maas-api-e2e-api-b-59a5a8-serving-cert" not found Sep 16 13:55:59.769218 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:55:59.769180 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a22fa4c0-cdee-42dd-ba5d-52ece500282c-maas-api-tls podName:a22fa4c0-cdee-42dd-ba5d-52ece500282c nodeName:}" failed. No retries permitted until 2026-09-16 13:56:00.26916191 +0000 UTC m=+1521.313464862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a22fa4c0-cdee-42dd-ba5d-52ece500282c-maas-api-tls") pod "maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" (UID: "a22fa4c0-cdee-42dd-ba5d-52ece500282c") : secret "maas-api-e2e-api-b-59a5a8-serving-cert" not found Sep 16 13:55:59.769603 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:59.769578 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22fa4c0-cdee-42dd-ba5d-52ece500282c-openshift-service-ca\") pod \"maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd\" (UID: \"a22fa4c0-cdee-42dd-ba5d-52ece500282c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" Sep 16 13:55:59.771499 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:59.771473 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22fa4c0-cdee-42dd-ba5d-52ece500282c-metrics-certs\") pod \"maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd\" (UID: \"a22fa4c0-cdee-42dd-ba5d-52ece500282c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" Sep 16 13:55:59.778141 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:55:59.778109 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf85p\" (UniqueName: \"kubernetes.io/projected/a22fa4c0-cdee-42dd-ba5d-52ece500282c-kube-api-access-sf85p\") pod \"maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd\" (UID: \"a22fa4c0-cdee-42dd-ba5d-52ece500282c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" Sep 16 13:56:00.271958 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:00.271923 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a22fa4c0-cdee-42dd-ba5d-52ece500282c-maas-api-tls\") pod \"maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd\" (UID: \"a22fa4c0-cdee-42dd-ba5d-52ece500282c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" Sep 16 13:56:00.274435 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:00.274415 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a22fa4c0-cdee-42dd-ba5d-52ece500282c-maas-api-tls\") pod \"maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd\" (UID: \"a22fa4c0-cdee-42dd-ba5d-52ece500282c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" Sep 16 13:56:00.528435 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:00.528349 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" Sep 16 13:56:00.803944 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:00.803916 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd"] Sep 16 13:56:00.805734 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:56:00.805704 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22fa4c0_cdee_42dd_ba5d_52ece500282c.slice/crio-46e41711b8f496d0f5821c19bf17c203f7d6831668c01b4fc4b0a3412edb8e6a WatchSource:0}: Error finding container 46e41711b8f496d0f5821c19bf17c203f7d6831668c01b4fc4b0a3412edb8e6a: Status 404 returned error can't find the container with id 46e41711b8f496d0f5821c19bf17c203f7d6831668c01b4fc4b0a3412edb8e6a Sep 16 13:56:01.010855 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:01.010821 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" event={"ID":"a22fa4c0-cdee-42dd-ba5d-52ece500282c","Type":"ContainerStarted","Data":"46e41711b8f496d0f5821c19bf17c203f7d6831668c01b4fc4b0a3412edb8e6a"} Sep 16 13:56:02.018034 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:02.018000 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" event={"ID":"a22fa4c0-cdee-42dd-ba5d-52ece500282c","Type":"ContainerStarted","Data":"fac8f1898c02cdabebd54e84b8005282d6beed909b90066d6ed0ad7062562a6b"} Sep 16 13:56:02.018446 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:02.018071 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" Sep 16 13:56:02.037433 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:02.037361 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" podStartSLOduration=2.5939857379999998 podStartE2EDuration="3.037343918s" totalImagesPullingTime="443.35818ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:56:02.036489065 +0000 UTC m=+1523.080792051" watchObservedRunningTime="2026-09-16 13:56:02.037343918 +0000 UTC m=+1523.081646884" Sep 16 13:56:03.522051 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:03.522015 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9"] Sep 16 13:56:03.529691 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:03.529658 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6"] Sep 16 13:56:08.031472 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:08.031446 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" Sep 16 13:56:14.444328 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:14.444293 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv"] Sep 16 13:56:14.444881 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:14.444567 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" podUID="4bbd6dfb-b359-433c-8cc2-3cc44869985b" containerName="maas-api" containerID="cri-o://dcf416d9844cde654a9c6a54c481df9848136c68931eca96516bf2608f4f1b54" gracePeriod=30 Sep 16 13:56:14.692662 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:14.692636 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" Sep 16 13:56:14.816643 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:14.816564 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbd6dfb-b359-433c-8cc2-3cc44869985b-metrics-certs\") pod \"4bbd6dfb-b359-433c-8cc2-3cc44869985b\" (UID: \"4bbd6dfb-b359-433c-8cc2-3cc44869985b\") " Sep 16 13:56:14.816643 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:14.816622 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bbd6dfb-b359-433c-8cc2-3cc44869985b-openshift-service-ca\") pod \"4bbd6dfb-b359-433c-8cc2-3cc44869985b\" (UID: \"4bbd6dfb-b359-433c-8cc2-3cc44869985b\") " Sep 16 13:56:14.816866 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:14.816679 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bbd6dfb-b359-433c-8cc2-3cc44869985b-maas-api-tls\") pod \"4bbd6dfb-b359-433c-8cc2-3cc44869985b\" (UID: \"4bbd6dfb-b359-433c-8cc2-3cc44869985b\") " Sep 16 13:56:14.816866 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:14.816709 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr6hf\" (UniqueName: \"kubernetes.io/projected/4bbd6dfb-b359-433c-8cc2-3cc44869985b-kube-api-access-wr6hf\") pod \"4bbd6dfb-b359-433c-8cc2-3cc44869985b\" (UID: \"4bbd6dfb-b359-433c-8cc2-3cc44869985b\") " Sep 16 13:56:14.817105 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:14.817071 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bbd6dfb-b359-433c-8cc2-3cc44869985b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4bbd6dfb-b359-433c-8cc2-3cc44869985b" (UID: "4bbd6dfb-b359-433c-8cc2-3cc44869985b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:56:14.819485 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:14.819457 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bbd6dfb-b359-433c-8cc2-3cc44869985b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4bbd6dfb-b359-433c-8cc2-3cc44869985b" (UID: "4bbd6dfb-b359-433c-8cc2-3cc44869985b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:56:14.819485 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:14.819473 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bbd6dfb-b359-433c-8cc2-3cc44869985b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4bbd6dfb-b359-433c-8cc2-3cc44869985b" (UID: "4bbd6dfb-b359-433c-8cc2-3cc44869985b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:56:14.819655 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:14.819590 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bbd6dfb-b359-433c-8cc2-3cc44869985b-kube-api-access-wr6hf" (OuterVolumeSpecName: "kube-api-access-wr6hf") pod "4bbd6dfb-b359-433c-8cc2-3cc44869985b" (UID: "4bbd6dfb-b359-433c-8cc2-3cc44869985b"). InnerVolumeSpecName "kube-api-access-wr6hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:56:14.918436 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:14.918405 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bbd6dfb-b359-433c-8cc2-3cc44869985b-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:56:14.918436 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:14.918430 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr6hf\" (UniqueName: \"kubernetes.io/projected/4bbd6dfb-b359-433c-8cc2-3cc44869985b-kube-api-access-wr6hf\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:56:14.918644 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:14.918447 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bbd6dfb-b359-433c-8cc2-3cc44869985b-metrics-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:56:14.918644 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:14.918455 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bbd6dfb-b359-433c-8cc2-3cc44869985b-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:56:15.102519 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:15.102434 2562 generic.go:356] "Generic (PLEG): container finished" podID="4bbd6dfb-b359-433c-8cc2-3cc44869985b" containerID="dcf416d9844cde654a9c6a54c481df9848136c68931eca96516bf2608f4f1b54" exitCode=0 Sep 16 13:56:15.102519 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:15.102482 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" Sep 16 13:56:15.102703 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:15.102519 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" event={"ID":"4bbd6dfb-b359-433c-8cc2-3cc44869985b","Type":"ContainerDied","Data":"dcf416d9844cde654a9c6a54c481df9848136c68931eca96516bf2608f4f1b54"} Sep 16 13:56:15.102703 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:15.102575 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv" event={"ID":"4bbd6dfb-b359-433c-8cc2-3cc44869985b","Type":"ContainerDied","Data":"84590811f3aea733b47a8979192c9f08ea8d99548ec02e568b5f9dfa67c97195"} Sep 16 13:56:15.102703 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:15.102617 2562 scope.go:117] "RemoveContainer" containerID="dcf416d9844cde654a9c6a54c481df9848136c68931eca96516bf2608f4f1b54" Sep 16 13:56:15.113428 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:15.113407 2562 scope.go:117] "RemoveContainer" containerID="dcf416d9844cde654a9c6a54c481df9848136c68931eca96516bf2608f4f1b54" Sep 16 13:56:15.113696 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:56:15.113678 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcf416d9844cde654a9c6a54c481df9848136c68931eca96516bf2608f4f1b54\": container with ID starting with dcf416d9844cde654a9c6a54c481df9848136c68931eca96516bf2608f4f1b54 not found: ID does not exist" containerID="dcf416d9844cde654a9c6a54c481df9848136c68931eca96516bf2608f4f1b54" Sep 16 13:56:15.113762 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:15.113707 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcf416d9844cde654a9c6a54c481df9848136c68931eca96516bf2608f4f1b54"} err="failed to get container status \"dcf416d9844cde654a9c6a54c481df9848136c68931eca96516bf2608f4f1b54\": rpc error: code = NotFound desc = could not find container \"dcf416d9844cde654a9c6a54c481df9848136c68931eca96516bf2608f4f1b54\": container with ID starting with dcf416d9844cde654a9c6a54c481df9848136c68931eca96516bf2608f4f1b54 not found: ID does not exist" Sep 16 13:56:15.138753 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:15.138723 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv"] Sep 16 13:56:15.141284 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:15.141261 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-46ccbd-697dc775dc-5gnsv"] Sep 16 13:56:15.414827 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:15.414797 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bbd6dfb-b359-433c-8cc2-3cc44869985b" path="/var/lib/kubelet/pods/4bbd6dfb-b359-433c-8cc2-3cc44869985b/volumes" Sep 16 13:56:18.542907 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.542853 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" podUID="13b282d7-4177-478e-b1de-49b9eb1da87b" containerName="main" containerID="cri-o://b31aebd9e1d927b484c79073aaa718c7300ed4730ff7a289e0765e57a578f96c" gracePeriod=45 Sep 16 13:56:18.549036 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.548999 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" podUID="df453e94-cc90-4e00-a3f0-7106c99da452" containerName="main" containerID="cri-o://e1178ae8e904ac6a6e9b2d77853bfd7f64a0740c54fc618ba910a969d68d0398" gracePeriod=45 Sep 16 13:56:18.802077 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.802012 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:56:18.837149 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.837122 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:56:18.861114 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.861088 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdr27\" (UniqueName: \"kubernetes.io/projected/13b282d7-4177-478e-b1de-49b9eb1da87b-kube-api-access-bdr27\") pod \"13b282d7-4177-478e-b1de-49b9eb1da87b\" (UID: \"13b282d7-4177-478e-b1de-49b9eb1da87b\") " Sep 16 13:56:18.861284 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.861149 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13b282d7-4177-478e-b1de-49b9eb1da87b-dshm\") pod \"13b282d7-4177-478e-b1de-49b9eb1da87b\" (UID: \"13b282d7-4177-478e-b1de-49b9eb1da87b\") " Sep 16 13:56:18.861284 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.861202 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13b282d7-4177-478e-b1de-49b9eb1da87b-model-cache\") pod \"13b282d7-4177-478e-b1de-49b9eb1da87b\" (UID: \"13b282d7-4177-478e-b1de-49b9eb1da87b\") " Sep 16 13:56:18.861284 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.861255 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13b282d7-4177-478e-b1de-49b9eb1da87b-home\") pod \"13b282d7-4177-478e-b1de-49b9eb1da87b\" (UID: \"13b282d7-4177-478e-b1de-49b9eb1da87b\") " Sep 16 13:56:18.861595 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.861350 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13b282d7-4177-478e-b1de-49b9eb1da87b-tmp-dir\") pod \"13b282d7-4177-478e-b1de-49b9eb1da87b\" (UID: \"13b282d7-4177-478e-b1de-49b9eb1da87b\") " Sep 16 13:56:18.861595 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.861392 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13b282d7-4177-478e-b1de-49b9eb1da87b-tls-certs\") pod \"13b282d7-4177-478e-b1de-49b9eb1da87b\" (UID: \"13b282d7-4177-478e-b1de-49b9eb1da87b\") " Sep 16 13:56:18.861719 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.861623 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b282d7-4177-478e-b1de-49b9eb1da87b-model-cache" (OuterVolumeSpecName: "model-cache") pod "13b282d7-4177-478e-b1de-49b9eb1da87b" (UID: "13b282d7-4177-478e-b1de-49b9eb1da87b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:18.861858 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.861837 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13b282d7-4177-478e-b1de-49b9eb1da87b-model-cache\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:56:18.861984 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.861918 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b282d7-4177-478e-b1de-49b9eb1da87b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13b282d7-4177-478e-b1de-49b9eb1da87b" (UID: "13b282d7-4177-478e-b1de-49b9eb1da87b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:18.862063 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.861980 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b282d7-4177-478e-b1de-49b9eb1da87b-home" (OuterVolumeSpecName: "home") pod "13b282d7-4177-478e-b1de-49b9eb1da87b" (UID: "13b282d7-4177-478e-b1de-49b9eb1da87b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:18.863838 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.863810 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b282d7-4177-478e-b1de-49b9eb1da87b-kube-api-access-bdr27" (OuterVolumeSpecName: "kube-api-access-bdr27") pod "13b282d7-4177-478e-b1de-49b9eb1da87b" (UID: "13b282d7-4177-478e-b1de-49b9eb1da87b"). InnerVolumeSpecName "kube-api-access-bdr27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:56:18.863929 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.863895 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13b282d7-4177-478e-b1de-49b9eb1da87b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13b282d7-4177-478e-b1de-49b9eb1da87b" (UID: "13b282d7-4177-478e-b1de-49b9eb1da87b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:56:18.864251 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.864236 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b282d7-4177-478e-b1de-49b9eb1da87b-dshm" (OuterVolumeSpecName: "dshm") pod "13b282d7-4177-478e-b1de-49b9eb1da87b" (UID: "13b282d7-4177-478e-b1de-49b9eb1da87b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:18.962537 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.962510 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkf6j\" (UniqueName: \"kubernetes.io/projected/df453e94-cc90-4e00-a3f0-7106c99da452-kube-api-access-vkf6j\") pod \"df453e94-cc90-4e00-a3f0-7106c99da452\" (UID: \"df453e94-cc90-4e00-a3f0-7106c99da452\") " Sep 16 13:56:18.962537 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.962540 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df453e94-cc90-4e00-a3f0-7106c99da452-tls-certs\") pod \"df453e94-cc90-4e00-a3f0-7106c99da452\" (UID: \"df453e94-cc90-4e00-a3f0-7106c99da452\") " Sep 16 13:56:18.962779 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.962582 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df453e94-cc90-4e00-a3f0-7106c99da452-dshm\") pod \"df453e94-cc90-4e00-a3f0-7106c99da452\" (UID: \"df453e94-cc90-4e00-a3f0-7106c99da452\") " Sep 16 13:56:18.962779 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.962667 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df453e94-cc90-4e00-a3f0-7106c99da452-model-cache\") pod \"df453e94-cc90-4e00-a3f0-7106c99da452\" (UID: \"df453e94-cc90-4e00-a3f0-7106c99da452\") " Sep 16 13:56:18.962779 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.962693 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df453e94-cc90-4e00-a3f0-7106c99da452-tmp-dir\") pod \"df453e94-cc90-4e00-a3f0-7106c99da452\" (UID: \"df453e94-cc90-4e00-a3f0-7106c99da452\") " Sep 16 13:56:18.962779 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.962731 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df453e94-cc90-4e00-a3f0-7106c99da452-home\") pod \"df453e94-cc90-4e00-a3f0-7106c99da452\" (UID: \"df453e94-cc90-4e00-a3f0-7106c99da452\") " Sep 16 13:56:18.963039 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.962968 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df453e94-cc90-4e00-a3f0-7106c99da452-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "df453e94-cc90-4e00-a3f0-7106c99da452" (UID: "df453e94-cc90-4e00-a3f0-7106c99da452"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:18.963121 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.963087 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df453e94-cc90-4e00-a3f0-7106c99da452-home" (OuterVolumeSpecName: "home") pod "df453e94-cc90-4e00-a3f0-7106c99da452" (UID: "df453e94-cc90-4e00-a3f0-7106c99da452"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:18.963203 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.963188 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df453e94-cc90-4e00-a3f0-7106c99da452-tmp-dir\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:56:18.963264 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.963210 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13b282d7-4177-478e-b1de-49b9eb1da87b-tmp-dir\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:56:18.963264 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.963222 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13b282d7-4177-478e-b1de-49b9eb1da87b-tls-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:56:18.963264 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.963239 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdr27\" (UniqueName: \"kubernetes.io/projected/13b282d7-4177-478e-b1de-49b9eb1da87b-kube-api-access-bdr27\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:56:18.963264 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.963252 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13b282d7-4177-478e-b1de-49b9eb1da87b-dshm\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:56:18.963443 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.963265 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13b282d7-4177-478e-b1de-49b9eb1da87b-home\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:56:18.963443 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.963365 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df453e94-cc90-4e00-a3f0-7106c99da452-model-cache" (OuterVolumeSpecName: "model-cache") pod "df453e94-cc90-4e00-a3f0-7106c99da452" (UID: "df453e94-cc90-4e00-a3f0-7106c99da452"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:18.964943 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.964913 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df453e94-cc90-4e00-a3f0-7106c99da452-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df453e94-cc90-4e00-a3f0-7106c99da452" (UID: "df453e94-cc90-4e00-a3f0-7106c99da452"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:56:18.965760 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.965739 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df453e94-cc90-4e00-a3f0-7106c99da452-dshm" (OuterVolumeSpecName: "dshm") pod "df453e94-cc90-4e00-a3f0-7106c99da452" (UID: "df453e94-cc90-4e00-a3f0-7106c99da452"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:56:18.965845 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:18.965768 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df453e94-cc90-4e00-a3f0-7106c99da452-kube-api-access-vkf6j" (OuterVolumeSpecName: "kube-api-access-vkf6j") pod "df453e94-cc90-4e00-a3f0-7106c99da452" (UID: "df453e94-cc90-4e00-a3f0-7106c99da452"). InnerVolumeSpecName "kube-api-access-vkf6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:56:19.064222 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.064130 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df453e94-cc90-4e00-a3f0-7106c99da452-home\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:56:19.064222 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.064160 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkf6j\" (UniqueName: \"kubernetes.io/projected/df453e94-cc90-4e00-a3f0-7106c99da452-kube-api-access-vkf6j\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:56:19.064222 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.064169 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df453e94-cc90-4e00-a3f0-7106c99da452-tls-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:56:19.064222 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.064176 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df453e94-cc90-4e00-a3f0-7106c99da452-dshm\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:56:19.064222 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.064184 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df453e94-cc90-4e00-a3f0-7106c99da452-model-cache\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:56:19.129111 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.129082 2562 generic.go:356] "Generic (PLEG): container finished" podID="df453e94-cc90-4e00-a3f0-7106c99da452" containerID="e1178ae8e904ac6a6e9b2d77853bfd7f64a0740c54fc618ba910a969d68d0398" exitCode=0 Sep 16 13:56:19.129306 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.129133 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" Sep 16 13:56:19.129306 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.129167 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" event={"ID":"df453e94-cc90-4e00-a3f0-7106c99da452","Type":"ContainerDied","Data":"e1178ae8e904ac6a6e9b2d77853bfd7f64a0740c54fc618ba910a969d68d0398"} Sep 16 13:56:19.129306 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.129220 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6" event={"ID":"df453e94-cc90-4e00-a3f0-7106c99da452","Type":"ContainerDied","Data":"697e61f0b5baec85802d2a554fb37fc80e1de0bf8c6e3386fc54eb94d1905a71"} Sep 16 13:56:19.129306 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.129243 2562 scope.go:117] "RemoveContainer" containerID="e1178ae8e904ac6a6e9b2d77853bfd7f64a0740c54fc618ba910a969d68d0398" Sep 16 13:56:19.130693 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.130673 2562 generic.go:356] "Generic (PLEG): container finished" podID="13b282d7-4177-478e-b1de-49b9eb1da87b" containerID="b31aebd9e1d927b484c79073aaa718c7300ed4730ff7a289e0765e57a578f96c" exitCode=0 Sep 16 13:56:19.130796 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.130704 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" event={"ID":"13b282d7-4177-478e-b1de-49b9eb1da87b","Type":"ContainerDied","Data":"b31aebd9e1d927b484c79073aaa718c7300ed4730ff7a289e0765e57a578f96c"} Sep 16 13:56:19.130796 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.130750 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" event={"ID":"13b282d7-4177-478e-b1de-49b9eb1da87b","Type":"ContainerDied","Data":"229c2b0c812322e04ed6a59f26710731343a5511fef5e44c22856af1f2e7c564"} Sep 16 13:56:19.130796 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.130718 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9" Sep 16 13:56:19.139445 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.139421 2562 scope.go:117] "RemoveContainer" containerID="e1178ae8e904ac6a6e9b2d77853bfd7f64a0740c54fc618ba910a969d68d0398" Sep 16 13:56:19.139800 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:56:19.139778 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1178ae8e904ac6a6e9b2d77853bfd7f64a0740c54fc618ba910a969d68d0398\": container with ID starting with e1178ae8e904ac6a6e9b2d77853bfd7f64a0740c54fc618ba910a969d68d0398 not found: ID does not exist" containerID="e1178ae8e904ac6a6e9b2d77853bfd7f64a0740c54fc618ba910a969d68d0398" Sep 16 13:56:19.139857 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.139808 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1178ae8e904ac6a6e9b2d77853bfd7f64a0740c54fc618ba910a969d68d0398"} err="failed to get container status \"e1178ae8e904ac6a6e9b2d77853bfd7f64a0740c54fc618ba910a969d68d0398\": rpc error: code = NotFound desc = could not find container \"e1178ae8e904ac6a6e9b2d77853bfd7f64a0740c54fc618ba910a969d68d0398\": container with ID starting with e1178ae8e904ac6a6e9b2d77853bfd7f64a0740c54fc618ba910a969d68d0398 not found: ID does not exist" Sep 16 13:56:19.139857 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.139824 2562 scope.go:117] "RemoveContainer" containerID="b31aebd9e1d927b484c79073aaa718c7300ed4730ff7a289e0765e57a578f96c" Sep 16 13:56:19.148473 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.148457 2562 scope.go:117] "RemoveContainer" containerID="b31aebd9e1d927b484c79073aaa718c7300ed4730ff7a289e0765e57a578f96c" Sep 16 13:56:19.148730 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:56:19.148713 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b31aebd9e1d927b484c79073aaa718c7300ed4730ff7a289e0765e57a578f96c\": container with ID starting with b31aebd9e1d927b484c79073aaa718c7300ed4730ff7a289e0765e57a578f96c not found: ID does not exist" containerID="b31aebd9e1d927b484c79073aaa718c7300ed4730ff7a289e0765e57a578f96c" Sep 16 13:56:19.148775 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.148737 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b31aebd9e1d927b484c79073aaa718c7300ed4730ff7a289e0765e57a578f96c"} err="failed to get container status \"b31aebd9e1d927b484c79073aaa718c7300ed4730ff7a289e0765e57a578f96c\": rpc error: code = NotFound desc = could not find container \"b31aebd9e1d927b484c79073aaa718c7300ed4730ff7a289e0765e57a578f96c\": container with ID starting with b31aebd9e1d927b484c79073aaa718c7300ed4730ff7a289e0765e57a578f96c not found: ID does not exist" Sep 16 13:56:19.162445 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.162422 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6"] Sep 16 13:56:19.165639 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.165616 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/embedding-w0-46ccbd-kserve-7bcc78f694-2hjb6"] Sep 16 13:56:19.174927 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.174896 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9"] Sep 16 13:56:19.177477 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.177455 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-46ccbd/distinct-w0-46ccbd-kserve-5bfcb774bf-w5rl9"] Sep 16 13:56:19.415226 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.415197 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13b282d7-4177-478e-b1de-49b9eb1da87b" path="/var/lib/kubelet/pods/13b282d7-4177-478e-b1de-49b9eb1da87b/volumes" Sep 16 13:56:19.415629 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:19.415616 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df453e94-cc90-4e00-a3f0-7106c99da452" path="/var/lib/kubelet/pods/df453e94-cc90-4e00-a3f0-7106c99da452/volumes" Sep 16 13:56:42.835200 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:42.835165 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr"] Sep 16 13:56:42.835666 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:42.835469 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" podUID="5427392c-91e4-45ea-bc64-9d9ca1f5b404" containerName="maas-api" containerID="cri-o://635d3d3fd43f5b6862cf2321ca436c9970442f8fdfa2fd22fd62d00f72034102" gracePeriod=30 Sep 16 13:56:43.092308 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:43.092236 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" Sep 16 13:56:43.208071 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:43.208034 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5427392c-91e4-45ea-bc64-9d9ca1f5b404-openshift-service-ca\") pod \"5427392c-91e4-45ea-bc64-9d9ca1f5b404\" (UID: \"5427392c-91e4-45ea-bc64-9d9ca1f5b404\") " Sep 16 13:56:43.208071 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:43.208079 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6r7mt\" (UniqueName: \"kubernetes.io/projected/5427392c-91e4-45ea-bc64-9d9ca1f5b404-kube-api-access-6r7mt\") pod \"5427392c-91e4-45ea-bc64-9d9ca1f5b404\" (UID: \"5427392c-91e4-45ea-bc64-9d9ca1f5b404\") " Sep 16 13:56:43.208367 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:43.208203 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5427392c-91e4-45ea-bc64-9d9ca1f5b404-maas-api-tls\") pod \"5427392c-91e4-45ea-bc64-9d9ca1f5b404\" (UID: \"5427392c-91e4-45ea-bc64-9d9ca1f5b404\") " Sep 16 13:56:43.208367 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:43.208244 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5427392c-91e4-45ea-bc64-9d9ca1f5b404-metrics-certs\") pod \"5427392c-91e4-45ea-bc64-9d9ca1f5b404\" (UID: \"5427392c-91e4-45ea-bc64-9d9ca1f5b404\") " Sep 16 13:56:43.208499 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:43.208466 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5427392c-91e4-45ea-bc64-9d9ca1f5b404-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5427392c-91e4-45ea-bc64-9d9ca1f5b404" (UID: "5427392c-91e4-45ea-bc64-9d9ca1f5b404"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:56:43.208697 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:43.208675 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5427392c-91e4-45ea-bc64-9d9ca1f5b404-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:56:43.210760 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:43.210721 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5427392c-91e4-45ea-bc64-9d9ca1f5b404-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5427392c-91e4-45ea-bc64-9d9ca1f5b404" (UID: "5427392c-91e4-45ea-bc64-9d9ca1f5b404"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:56:43.210869 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:43.210755 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5427392c-91e4-45ea-bc64-9d9ca1f5b404-kube-api-access-6r7mt" (OuterVolumeSpecName: "kube-api-access-6r7mt") pod "5427392c-91e4-45ea-bc64-9d9ca1f5b404" (UID: "5427392c-91e4-45ea-bc64-9d9ca1f5b404"). InnerVolumeSpecName "kube-api-access-6r7mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:56:43.210869 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:43.210802 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5427392c-91e4-45ea-bc64-9d9ca1f5b404-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5427392c-91e4-45ea-bc64-9d9ca1f5b404" (UID: "5427392c-91e4-45ea-bc64-9d9ca1f5b404"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:56:43.285118 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:43.285085 2562 generic.go:356] "Generic (PLEG): container finished" podID="5427392c-91e4-45ea-bc64-9d9ca1f5b404" containerID="635d3d3fd43f5b6862cf2321ca436c9970442f8fdfa2fd22fd62d00f72034102" exitCode=0 Sep 16 13:56:43.285329 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:43.285137 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" Sep 16 13:56:43.285329 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:43.285169 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" event={"ID":"5427392c-91e4-45ea-bc64-9d9ca1f5b404","Type":"ContainerDied","Data":"635d3d3fd43f5b6862cf2321ca436c9970442f8fdfa2fd22fd62d00f72034102"} Sep 16 13:56:43.285329 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:43.285232 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr" event={"ID":"5427392c-91e4-45ea-bc64-9d9ca1f5b404","Type":"ContainerDied","Data":"ff76771f20d8b23a0df12de376c3a97d9e83198441c02f840a9a968934378f05"} Sep 16 13:56:43.285329 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:43.285263 2562 scope.go:117] "RemoveContainer" containerID="635d3d3fd43f5b6862cf2321ca436c9970442f8fdfa2fd22fd62d00f72034102" Sep 16 13:56:43.295314 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:43.295298 2562 scope.go:117] "RemoveContainer" containerID="635d3d3fd43f5b6862cf2321ca436c9970442f8fdfa2fd22fd62d00f72034102" Sep 16 13:56:43.295613 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:56:43.295594 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"635d3d3fd43f5b6862cf2321ca436c9970442f8fdfa2fd22fd62d00f72034102\": container with ID starting with 635d3d3fd43f5b6862cf2321ca436c9970442f8fdfa2fd22fd62d00f72034102 not found: ID does not exist" containerID="635d3d3fd43f5b6862cf2321ca436c9970442f8fdfa2fd22fd62d00f72034102" Sep 16 13:56:43.295669 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:43.295621 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"635d3d3fd43f5b6862cf2321ca436c9970442f8fdfa2fd22fd62d00f72034102"} err="failed to get container status \"635d3d3fd43f5b6862cf2321ca436c9970442f8fdfa2fd22fd62d00f72034102\": rpc error: code = NotFound desc = could not find container \"635d3d3fd43f5b6862cf2321ca436c9970442f8fdfa2fd22fd62d00f72034102\": container with ID starting with 635d3d3fd43f5b6862cf2321ca436c9970442f8fdfa2fd22fd62d00f72034102 not found: ID does not exist" Sep 16 13:56:43.309620 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:43.309594 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr"] Sep 16 13:56:43.309926 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:43.309911 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5427392c-91e4-45ea-bc64-9d9ca1f5b404-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:56:43.309973 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:43.309929 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5427392c-91e4-45ea-bc64-9d9ca1f5b404-metrics-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:56:43.309973 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:43.309938 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r7mt\" (UniqueName: \"kubernetes.io/projected/5427392c-91e4-45ea-bc64-9d9ca1f5b404-kube-api-access-6r7mt\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:56:43.316305 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:43.316287 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bdd04e-6bd6df4bfd-8lgqr"] Sep 16 13:56:43.413813 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:56:43.413782 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5427392c-91e4-45ea-bc64-9d9ca1f5b404" path="/var/lib/kubelet/pods/5427392c-91e4-45ea-bc64-9d9ca1f5b404/volumes" Sep 16 13:57:10.993662 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:10.993627 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd"] Sep 16 13:57:10.994713 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:10.993873 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" podUID="a22fa4c0-cdee-42dd-ba5d-52ece500282c" containerName="maas-api" containerID="cri-o://fac8f1898c02cdabebd54e84b8005282d6beed909b90066d6ed0ad7062562a6b" gracePeriod=30 Sep 16 13:57:11.235088 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:11.235064 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" Sep 16 13:57:11.288280 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:11.288201 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22fa4c0-cdee-42dd-ba5d-52ece500282c-openshift-service-ca\") pod \"a22fa4c0-cdee-42dd-ba5d-52ece500282c\" (UID: \"a22fa4c0-cdee-42dd-ba5d-52ece500282c\") " Sep 16 13:57:11.288280 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:11.288256 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22fa4c0-cdee-42dd-ba5d-52ece500282c-metrics-certs\") pod \"a22fa4c0-cdee-42dd-ba5d-52ece500282c\" (UID: \"a22fa4c0-cdee-42dd-ba5d-52ece500282c\") " Sep 16 13:57:11.288522 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:11.288292 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a22fa4c0-cdee-42dd-ba5d-52ece500282c-maas-api-tls\") pod \"a22fa4c0-cdee-42dd-ba5d-52ece500282c\" (UID: \"a22fa4c0-cdee-42dd-ba5d-52ece500282c\") " Sep 16 13:57:11.288522 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:11.288479 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sf85p\" (UniqueName: \"kubernetes.io/projected/a22fa4c0-cdee-42dd-ba5d-52ece500282c-kube-api-access-sf85p\") pod \"a22fa4c0-cdee-42dd-ba5d-52ece500282c\" (UID: \"a22fa4c0-cdee-42dd-ba5d-52ece500282c\") " Sep 16 13:57:11.288660 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:11.288640 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22fa4c0-cdee-42dd-ba5d-52ece500282c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a22fa4c0-cdee-42dd-ba5d-52ece500282c" (UID: "a22fa4c0-cdee-42dd-ba5d-52ece500282c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:57:11.288784 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:11.288765 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22fa4c0-cdee-42dd-ba5d-52ece500282c-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:11.290584 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:11.290558 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a22fa4c0-cdee-42dd-ba5d-52ece500282c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a22fa4c0-cdee-42dd-ba5d-52ece500282c" (UID: "a22fa4c0-cdee-42dd-ba5d-52ece500282c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:57:11.290686 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:11.290604 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22fa4c0-cdee-42dd-ba5d-52ece500282c-kube-api-access-sf85p" (OuterVolumeSpecName: "kube-api-access-sf85p") pod "a22fa4c0-cdee-42dd-ba5d-52ece500282c" (UID: "a22fa4c0-cdee-42dd-ba5d-52ece500282c"). InnerVolumeSpecName "kube-api-access-sf85p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:57:11.290686 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:11.290657 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a22fa4c0-cdee-42dd-ba5d-52ece500282c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a22fa4c0-cdee-42dd-ba5d-52ece500282c" (UID: "a22fa4c0-cdee-42dd-ba5d-52ece500282c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:57:11.390449 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:11.390416 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22fa4c0-cdee-42dd-ba5d-52ece500282c-metrics-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:11.390449 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:11.390442 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a22fa4c0-cdee-42dd-ba5d-52ece500282c-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:11.390449 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:11.390451 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf85p\" (UniqueName: \"kubernetes.io/projected/a22fa4c0-cdee-42dd-ba5d-52ece500282c-kube-api-access-sf85p\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:11.453666 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:11.453632 2562 generic.go:356] "Generic (PLEG): container finished" podID="a22fa4c0-cdee-42dd-ba5d-52ece500282c" containerID="fac8f1898c02cdabebd54e84b8005282d6beed909b90066d6ed0ad7062562a6b" exitCode=0 Sep 16 13:57:11.453853 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:11.453690 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" Sep 16 13:57:11.453853 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:11.453715 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" event={"ID":"a22fa4c0-cdee-42dd-ba5d-52ece500282c","Type":"ContainerDied","Data":"fac8f1898c02cdabebd54e84b8005282d6beed909b90066d6ed0ad7062562a6b"} Sep 16 13:57:11.453853 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:11.453771 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd" event={"ID":"a22fa4c0-cdee-42dd-ba5d-52ece500282c","Type":"ContainerDied","Data":"46e41711b8f496d0f5821c19bf17c203f7d6831668c01b4fc4b0a3412edb8e6a"} Sep 16 13:57:11.453853 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:11.453798 2562 scope.go:117] "RemoveContainer" containerID="fac8f1898c02cdabebd54e84b8005282d6beed909b90066d6ed0ad7062562a6b" Sep 16 13:57:11.463429 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:11.463408 2562 scope.go:117] "RemoveContainer" containerID="fac8f1898c02cdabebd54e84b8005282d6beed909b90066d6ed0ad7062562a6b" Sep 16 13:57:11.463722 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:57:11.463704 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac8f1898c02cdabebd54e84b8005282d6beed909b90066d6ed0ad7062562a6b\": container with ID starting with fac8f1898c02cdabebd54e84b8005282d6beed909b90066d6ed0ad7062562a6b not found: ID does not exist" containerID="fac8f1898c02cdabebd54e84b8005282d6beed909b90066d6ed0ad7062562a6b" Sep 16 13:57:11.463777 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:11.463731 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac8f1898c02cdabebd54e84b8005282d6beed909b90066d6ed0ad7062562a6b"} err="failed to get container status \"fac8f1898c02cdabebd54e84b8005282d6beed909b90066d6ed0ad7062562a6b\": rpc error: code = NotFound desc = could not find container \"fac8f1898c02cdabebd54e84b8005282d6beed909b90066d6ed0ad7062562a6b\": container with ID starting with fac8f1898c02cdabebd54e84b8005282d6beed909b90066d6ed0ad7062562a6b not found: ID does not exist" Sep 16 13:57:11.472606 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:11.472580 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd"] Sep 16 13:57:11.475649 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:11.475630 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-59a5a8-6584ff4f6c-hwckd"] Sep 16 13:57:12.468224 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:12.468189 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s"] Sep 16 13:57:12.468666 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:12.468397 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" podUID="37a19ec4-2357-4035-888e-505953c31e3e" containerName="maas-api" containerID="cri-o://eb4c633e67d7b48cae520587b9c5e2ae30ca18c729a2f90a718a2cdf36df123e" gracePeriod=30 Sep 16 13:57:12.537906 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:12.537693 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" podUID="37a19ec4-2357-4035-888e-505953c31e3e" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.54:8443/health\": dial tcp 10.132.0.54:8443: connect: connection refused" Sep 16 13:57:12.731315 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:12.731245 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" Sep 16 13:57:12.805070 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:12.805041 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37a19ec4-2357-4035-888e-505953c31e3e-openshift-service-ca\") pod \"37a19ec4-2357-4035-888e-505953c31e3e\" (UID: \"37a19ec4-2357-4035-888e-505953c31e3e\") " Sep 16 13:57:12.805262 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:12.805123 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37a19ec4-2357-4035-888e-505953c31e3e-maas-api-tls\") pod \"37a19ec4-2357-4035-888e-505953c31e3e\" (UID: \"37a19ec4-2357-4035-888e-505953c31e3e\") " Sep 16 13:57:12.805337 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:12.805281 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dt7jr\" (UniqueName: \"kubernetes.io/projected/37a19ec4-2357-4035-888e-505953c31e3e-kube-api-access-dt7jr\") pod \"37a19ec4-2357-4035-888e-505953c31e3e\" (UID: \"37a19ec4-2357-4035-888e-505953c31e3e\") " Sep 16 13:57:12.805337 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:12.805320 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37a19ec4-2357-4035-888e-505953c31e3e-metrics-certs\") pod \"37a19ec4-2357-4035-888e-505953c31e3e\" (UID: \"37a19ec4-2357-4035-888e-505953c31e3e\") " Sep 16 13:57:12.805487 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:12.805435 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37a19ec4-2357-4035-888e-505953c31e3e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "37a19ec4-2357-4035-888e-505953c31e3e" (UID: "37a19ec4-2357-4035-888e-505953c31e3e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:57:12.805683 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:12.805664 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37a19ec4-2357-4035-888e-505953c31e3e-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:12.807421 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:12.807353 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37a19ec4-2357-4035-888e-505953c31e3e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "37a19ec4-2357-4035-888e-505953c31e3e" (UID: "37a19ec4-2357-4035-888e-505953c31e3e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:57:12.807646 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:12.807555 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a19ec4-2357-4035-888e-505953c31e3e-kube-api-access-dt7jr" (OuterVolumeSpecName: "kube-api-access-dt7jr") pod "37a19ec4-2357-4035-888e-505953c31e3e" (UID: "37a19ec4-2357-4035-888e-505953c31e3e"). InnerVolumeSpecName "kube-api-access-dt7jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:57:12.807646 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:12.807620 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37a19ec4-2357-4035-888e-505953c31e3e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "37a19ec4-2357-4035-888e-505953c31e3e" (UID: "37a19ec4-2357-4035-888e-505953c31e3e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:57:12.906432 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:12.906392 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt7jr\" (UniqueName: \"kubernetes.io/projected/37a19ec4-2357-4035-888e-505953c31e3e-kube-api-access-dt7jr\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:12.906432 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:12.906427 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37a19ec4-2357-4035-888e-505953c31e3e-metrics-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:12.906432 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:12.906436 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37a19ec4-2357-4035-888e-505953c31e3e-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:13.413674 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:13.413642 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a22fa4c0-cdee-42dd-ba5d-52ece500282c" path="/var/lib/kubelet/pods/a22fa4c0-cdee-42dd-ba5d-52ece500282c/volumes" Sep 16 13:57:13.474102 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:13.474071 2562 generic.go:356] "Generic (PLEG): container finished" podID="37a19ec4-2357-4035-888e-505953c31e3e" containerID="eb4c633e67d7b48cae520587b9c5e2ae30ca18c729a2f90a718a2cdf36df123e" exitCode=0 Sep 16 13:57:13.474548 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:13.474123 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" Sep 16 13:57:13.474548 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:13.474145 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" event={"ID":"37a19ec4-2357-4035-888e-505953c31e3e","Type":"ContainerDied","Data":"eb4c633e67d7b48cae520587b9c5e2ae30ca18c729a2f90a718a2cdf36df123e"} Sep 16 13:57:13.474548 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:13.474200 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s" event={"ID":"37a19ec4-2357-4035-888e-505953c31e3e","Type":"ContainerDied","Data":"8712a29f6afbdf387c3ef6f6445f0aa656a034f25b9edbd7b2ec644a419e9641"} Sep 16 13:57:13.474548 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:13.474222 2562 scope.go:117] "RemoveContainer" containerID="eb4c633e67d7b48cae520587b9c5e2ae30ca18c729a2f90a718a2cdf36df123e" Sep 16 13:57:13.483480 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:13.483465 2562 scope.go:117] "RemoveContainer" containerID="eb4c633e67d7b48cae520587b9c5e2ae30ca18c729a2f90a718a2cdf36df123e" Sep 16 13:57:13.483752 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:57:13.483734 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb4c633e67d7b48cae520587b9c5e2ae30ca18c729a2f90a718a2cdf36df123e\": container with ID starting with eb4c633e67d7b48cae520587b9c5e2ae30ca18c729a2f90a718a2cdf36df123e not found: ID does not exist" containerID="eb4c633e67d7b48cae520587b9c5e2ae30ca18c729a2f90a718a2cdf36df123e" Sep 16 13:57:13.483801 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:13.483762 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4c633e67d7b48cae520587b9c5e2ae30ca18c729a2f90a718a2cdf36df123e"} err="failed to get container status \"eb4c633e67d7b48cae520587b9c5e2ae30ca18c729a2f90a718a2cdf36df123e\": rpc error: code = NotFound desc = could not find container \"eb4c633e67d7b48cae520587b9c5e2ae30ca18c729a2f90a718a2cdf36df123e\": container with ID starting with eb4c633e67d7b48cae520587b9c5e2ae30ca18c729a2f90a718a2cdf36df123e not found: ID does not exist" Sep 16 13:57:13.498698 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:13.498667 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s"] Sep 16 13:57:13.504498 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:13.504479 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4bb29c-6cd6bd4687-22t9s"] Sep 16 13:57:15.414559 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:15.414530 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37a19ec4-2357-4035-888e-505953c31e3e" path="/var/lib/kubelet/pods/37a19ec4-2357-4035-888e-505953c31e3e/volumes" Sep 16 13:57:20.225953 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:20.225916 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl"] Sep 16 13:57:20.226583 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:20.226220 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" podUID="6aefc8e6-56eb-46a4-b9a5-129a198815b7" containerName="istio-proxy" containerID="cri-o://dde57b1e1376160e38f4c9ac8607ce7ea6bc6a8d0e09e7c718ea5f06cb8ebd96" gracePeriod=30 Sep 16 13:57:25.477061 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.477032 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:57:25.528452 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.528362 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istio-envoy\") pod \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " Sep 16 13:57:25.528452 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.528419 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istio-podinfo\") pod \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " Sep 16 13:57:25.528668 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.528481 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istio-token\") pod \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " Sep 16 13:57:25.528668 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.528508 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-credential-socket\") pod \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " Sep 16 13:57:25.528668 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.528541 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istio-data\") pod \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " Sep 16 13:57:25.528668 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.528567 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-workload-socket\") pod \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " Sep 16 13:57:25.528668 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.528604 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istiod-ca-cert\") pod \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " Sep 16 13:57:25.528668 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.528643 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwcm5\" (UniqueName: \"kubernetes.io/projected/6aefc8e6-56eb-46a4-b9a5-129a198815b7-kube-api-access-nwcm5\") pod \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " Sep 16 13:57:25.528985 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.528693 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-workload-certs\") pod \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\" (UID: \"6aefc8e6-56eb-46a4-b9a5-129a198815b7\") " Sep 16 13:57:25.528985 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.528930 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6aefc8e6-56eb-46a4-b9a5-129a198815b7" (UID: "6aefc8e6-56eb-46a4-b9a5-129a198815b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:25.529393 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.529323 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-workload-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:25.529518 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.529440 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istio-data" (OuterVolumeSpecName: "istio-data") pod "6aefc8e6-56eb-46a4-b9a5-129a198815b7" (UID: "6aefc8e6-56eb-46a4-b9a5-129a198815b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:25.529584 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.529540 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6aefc8e6-56eb-46a4-b9a5-129a198815b7" (UID: "6aefc8e6-56eb-46a4-b9a5-129a198815b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:25.529753 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.529725 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6aefc8e6-56eb-46a4-b9a5-129a198815b7" (UID: "6aefc8e6-56eb-46a4-b9a5-129a198815b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:25.529922 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.529801 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6aefc8e6-56eb-46a4-b9a5-129a198815b7" (UID: "6aefc8e6-56eb-46a4-b9a5-129a198815b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:57:25.530917 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.530891 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6aefc8e6-56eb-46a4-b9a5-129a198815b7" (UID: "6aefc8e6-56eb-46a4-b9a5-129a198815b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 13:57:25.531006 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.530946 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istio-token" (OuterVolumeSpecName: "istio-token") pod "6aefc8e6-56eb-46a4-b9a5-129a198815b7" (UID: "6aefc8e6-56eb-46a4-b9a5-129a198815b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:57:25.531049 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.531023 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6aefc8e6-56eb-46a4-b9a5-129a198815b7" (UID: "6aefc8e6-56eb-46a4-b9a5-129a198815b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 13:57:25.531892 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.531873 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aefc8e6-56eb-46a4-b9a5-129a198815b7-kube-api-access-nwcm5" (OuterVolumeSpecName: "kube-api-access-nwcm5") pod "6aefc8e6-56eb-46a4-b9a5-129a198815b7" (UID: "6aefc8e6-56eb-46a4-b9a5-129a198815b7"). InnerVolumeSpecName "kube-api-access-nwcm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:57:25.549517 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.549493 2562 generic.go:356] "Generic (PLEG): container finished" podID="6aefc8e6-56eb-46a4-b9a5-129a198815b7" containerID="dde57b1e1376160e38f4c9ac8607ce7ea6bc6a8d0e09e7c718ea5f06cb8ebd96" exitCode=0 Sep 16 13:57:25.549617 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.549545 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" Sep 16 13:57:25.549617 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.549570 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" event={"ID":"6aefc8e6-56eb-46a4-b9a5-129a198815b7","Type":"ContainerDied","Data":"dde57b1e1376160e38f4c9ac8607ce7ea6bc6a8d0e09e7c718ea5f06cb8ebd96"} Sep 16 13:57:25.549735 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.549624 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl" event={"ID":"6aefc8e6-56eb-46a4-b9a5-129a198815b7","Type":"ContainerDied","Data":"dbac988688fbbdef31956fbd4d10e365f7efc4305ba1d1e79d96d5ff3acaa94b"} Sep 16 13:57:25.549735 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.549647 2562 scope.go:117] "RemoveContainer" containerID="dde57b1e1376160e38f4c9ac8607ce7ea6bc6a8d0e09e7c718ea5f06cb8ebd96" Sep 16 13:57:25.561278 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.561264 2562 scope.go:117] "RemoveContainer" containerID="423a591799bb20087eb6a2209703e5646d22f76d2c3f90099f881ddd0350eba5" Sep 16 13:57:25.569650 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.569632 2562 scope.go:117] "RemoveContainer" containerID="dde57b1e1376160e38f4c9ac8607ce7ea6bc6a8d0e09e7c718ea5f06cb8ebd96" Sep 16 13:57:25.569905 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:57:25.569884 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dde57b1e1376160e38f4c9ac8607ce7ea6bc6a8d0e09e7c718ea5f06cb8ebd96\": container with ID starting with dde57b1e1376160e38f4c9ac8607ce7ea6bc6a8d0e09e7c718ea5f06cb8ebd96 not found: ID does not exist" containerID="dde57b1e1376160e38f4c9ac8607ce7ea6bc6a8d0e09e7c718ea5f06cb8ebd96" Sep 16 13:57:25.569979 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.569918 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dde57b1e1376160e38f4c9ac8607ce7ea6bc6a8d0e09e7c718ea5f06cb8ebd96"} err="failed to get container status \"dde57b1e1376160e38f4c9ac8607ce7ea6bc6a8d0e09e7c718ea5f06cb8ebd96\": rpc error: code = NotFound desc = could not find container \"dde57b1e1376160e38f4c9ac8607ce7ea6bc6a8d0e09e7c718ea5f06cb8ebd96\": container with ID starting with dde57b1e1376160e38f4c9ac8607ce7ea6bc6a8d0e09e7c718ea5f06cb8ebd96 not found: ID does not exist" Sep 16 13:57:25.569979 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.569942 2562 scope.go:117] "RemoveContainer" containerID="423a591799bb20087eb6a2209703e5646d22f76d2c3f90099f881ddd0350eba5" Sep 16 13:57:25.570195 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:57:25.570178 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"423a591799bb20087eb6a2209703e5646d22f76d2c3f90099f881ddd0350eba5\": container with ID starting with 423a591799bb20087eb6a2209703e5646d22f76d2c3f90099f881ddd0350eba5 not found: ID does not exist" containerID="423a591799bb20087eb6a2209703e5646d22f76d2c3f90099f881ddd0350eba5" Sep 16 13:57:25.570239 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.570201 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"423a591799bb20087eb6a2209703e5646d22f76d2c3f90099f881ddd0350eba5"} err="failed to get container status \"423a591799bb20087eb6a2209703e5646d22f76d2c3f90099f881ddd0350eba5\": rpc error: code = NotFound desc = could not find container \"423a591799bb20087eb6a2209703e5646d22f76d2c3f90099f881ddd0350eba5\": container with ID starting with 423a591799bb20087eb6a2209703e5646d22f76d2c3f90099f881ddd0350eba5 not found: ID does not exist" Sep 16 13:57:25.575288 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.575269 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl"] Sep 16 13:57:25.579096 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.579077 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4bb29c-openshift-default-688994996-5vzvl"] Sep 16 13:57:25.630665 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.630639 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istiod-ca-cert\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:25.630665 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.630666 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwcm5\" (UniqueName: \"kubernetes.io/projected/6aefc8e6-56eb-46a4-b9a5-129a198815b7-kube-api-access-nwcm5\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:25.630832 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.630675 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istio-envoy\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:25.630832 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.630685 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istio-podinfo\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:25.630832 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.630692 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istio-token\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:25.630832 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.630700 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-credential-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:25.630832 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.630708 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-istio-data\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:25.630832 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:25.630716 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aefc8e6-56eb-46a4-b9a5-129a198815b7-workload-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:27.414036 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:27.413994 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6aefc8e6-56eb-46a4-b9a5-129a198815b7" path="/var/lib/kubelet/pods/6aefc8e6-56eb-46a4-b9a5-129a198815b7/volumes" Sep 16 13:57:28.734762 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:28.734728 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796"] Sep 16 13:57:28.735124 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:28.734995 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" podUID="741e39ff-8c9e-46da-b9b2-6ae7e259825f" containerName="maas-api" containerID="cri-o://8957e968759aebb45c64fa1437bc91e7bc0fe3c5d86973d90db545dddc2cd614" gracePeriod=30 Sep 16 13:57:28.984222 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:28.984194 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" Sep 16 13:57:29.065761 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:29.065682 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmw69\" (UniqueName: \"kubernetes.io/projected/741e39ff-8c9e-46da-b9b2-6ae7e259825f-kube-api-access-tmw69\") pod \"741e39ff-8c9e-46da-b9b2-6ae7e259825f\" (UID: \"741e39ff-8c9e-46da-b9b2-6ae7e259825f\") " Sep 16 13:57:29.065761 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:29.065744 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/741e39ff-8c9e-46da-b9b2-6ae7e259825f-openshift-service-ca\") pod \"741e39ff-8c9e-46da-b9b2-6ae7e259825f\" (UID: \"741e39ff-8c9e-46da-b9b2-6ae7e259825f\") " Sep 16 13:57:29.065955 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:29.065769 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/741e39ff-8c9e-46da-b9b2-6ae7e259825f-maas-api-tls\") pod \"741e39ff-8c9e-46da-b9b2-6ae7e259825f\" (UID: \"741e39ff-8c9e-46da-b9b2-6ae7e259825f\") " Sep 16 13:57:29.065955 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:29.065826 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/741e39ff-8c9e-46da-b9b2-6ae7e259825f-metrics-certs\") pod \"741e39ff-8c9e-46da-b9b2-6ae7e259825f\" (UID: \"741e39ff-8c9e-46da-b9b2-6ae7e259825f\") " Sep 16 13:57:29.066145 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:29.066114 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741e39ff-8c9e-46da-b9b2-6ae7e259825f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "741e39ff-8c9e-46da-b9b2-6ae7e259825f" (UID: "741e39ff-8c9e-46da-b9b2-6ae7e259825f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:57:29.068158 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:29.068135 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/741e39ff-8c9e-46da-b9b2-6ae7e259825f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "741e39ff-8c9e-46da-b9b2-6ae7e259825f" (UID: "741e39ff-8c9e-46da-b9b2-6ae7e259825f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:57:29.068158 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:29.068145 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741e39ff-8c9e-46da-b9b2-6ae7e259825f-kube-api-access-tmw69" (OuterVolumeSpecName: "kube-api-access-tmw69") pod "741e39ff-8c9e-46da-b9b2-6ae7e259825f" (UID: "741e39ff-8c9e-46da-b9b2-6ae7e259825f"). InnerVolumeSpecName "kube-api-access-tmw69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:57:29.068287 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:29.068156 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/741e39ff-8c9e-46da-b9b2-6ae7e259825f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "741e39ff-8c9e-46da-b9b2-6ae7e259825f" (UID: "741e39ff-8c9e-46da-b9b2-6ae7e259825f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:57:29.167286 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:29.167250 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/741e39ff-8c9e-46da-b9b2-6ae7e259825f-metrics-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:29.167286 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:29.167281 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmw69\" (UniqueName: \"kubernetes.io/projected/741e39ff-8c9e-46da-b9b2-6ae7e259825f-kube-api-access-tmw69\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:29.167286 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:29.167293 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/741e39ff-8c9e-46da-b9b2-6ae7e259825f-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:29.167555 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:29.167307 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/741e39ff-8c9e-46da-b9b2-6ae7e259825f-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:29.574954 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:29.574924 2562 generic.go:356] "Generic (PLEG): container finished" podID="741e39ff-8c9e-46da-b9b2-6ae7e259825f" containerID="8957e968759aebb45c64fa1437bc91e7bc0fe3c5d86973d90db545dddc2cd614" exitCode=0 Sep 16 13:57:29.575141 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:29.574962 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" Sep 16 13:57:29.575141 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:29.574998 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" event={"ID":"741e39ff-8c9e-46da-b9b2-6ae7e259825f","Type":"ContainerDied","Data":"8957e968759aebb45c64fa1437bc91e7bc0fe3c5d86973d90db545dddc2cd614"} Sep 16 13:57:29.575141 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:29.575041 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796" event={"ID":"741e39ff-8c9e-46da-b9b2-6ae7e259825f","Type":"ContainerDied","Data":"fabb6c12279d8b5de3a87407a9d991683341f7062e7cda015a98814bed2ce407"} Sep 16 13:57:29.575141 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:29.575066 2562 scope.go:117] "RemoveContainer" containerID="8957e968759aebb45c64fa1437bc91e7bc0fe3c5d86973d90db545dddc2cd614" Sep 16 13:57:29.584412 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:29.584365 2562 scope.go:117] "RemoveContainer" containerID="8957e968759aebb45c64fa1437bc91e7bc0fe3c5d86973d90db545dddc2cd614" Sep 16 13:57:29.584711 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:57:29.584690 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8957e968759aebb45c64fa1437bc91e7bc0fe3c5d86973d90db545dddc2cd614\": container with ID starting with 8957e968759aebb45c64fa1437bc91e7bc0fe3c5d86973d90db545dddc2cd614 not found: ID does not exist" containerID="8957e968759aebb45c64fa1437bc91e7bc0fe3c5d86973d90db545dddc2cd614" Sep 16 13:57:29.584791 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:29.584717 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8957e968759aebb45c64fa1437bc91e7bc0fe3c5d86973d90db545dddc2cd614"} err="failed to get container status \"8957e968759aebb45c64fa1437bc91e7bc0fe3c5d86973d90db545dddc2cd614\": rpc error: code = NotFound desc = could not find container \"8957e968759aebb45c64fa1437bc91e7bc0fe3c5d86973d90db545dddc2cd614\": container with ID starting with 8957e968759aebb45c64fa1437bc91e7bc0fe3c5d86973d90db545dddc2cd614 not found: ID does not exist" Sep 16 13:57:29.594836 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:29.594818 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796"] Sep 16 13:57:29.598902 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:29.598882 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-aff7af-588d66db65-qw796"] Sep 16 13:57:31.413593 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:31.413520 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="741e39ff-8c9e-46da-b9b2-6ae7e259825f" path="/var/lib/kubelet/pods/741e39ff-8c9e-46da-b9b2-6ae7e259825f/volumes" Sep 16 13:57:47.165865 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:47.165831 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx"] Sep 16 13:57:47.166306 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:47.166070 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx" podUID="5bae6be5-baf4-4e43-960a-24c42e43a1ad" containerName="maas-api" containerID="cri-o://79a3d3d1f10054985afb9a6c890061e4927165d37ba5b8770e63c417e6a1a1e9" gracePeriod=30 Sep 16 13:57:47.439935 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:47.439905 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx" Sep 16 13:57:47.538163 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:47.538131 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bae6be5-baf4-4e43-960a-24c42e43a1ad-openshift-service-ca\") pod \"5bae6be5-baf4-4e43-960a-24c42e43a1ad\" (UID: \"5bae6be5-baf4-4e43-960a-24c42e43a1ad\") " Sep 16 13:57:47.538341 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:47.538187 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bae6be5-baf4-4e43-960a-24c42e43a1ad-metrics-certs\") pod \"5bae6be5-baf4-4e43-960a-24c42e43a1ad\" (UID: \"5bae6be5-baf4-4e43-960a-24c42e43a1ad\") " Sep 16 13:57:47.538341 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:47.538227 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bae6be5-baf4-4e43-960a-24c42e43a1ad-maas-api-tls\") pod \"5bae6be5-baf4-4e43-960a-24c42e43a1ad\" (UID: \"5bae6be5-baf4-4e43-960a-24c42e43a1ad\") " Sep 16 13:57:47.538341 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:47.538254 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlhjc\" (UniqueName: \"kubernetes.io/projected/5bae6be5-baf4-4e43-960a-24c42e43a1ad-kube-api-access-vlhjc\") pod \"5bae6be5-baf4-4e43-960a-24c42e43a1ad\" (UID: \"5bae6be5-baf4-4e43-960a-24c42e43a1ad\") " Sep 16 13:57:47.538626 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:47.538586 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bae6be5-baf4-4e43-960a-24c42e43a1ad-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5bae6be5-baf4-4e43-960a-24c42e43a1ad" (UID: "5bae6be5-baf4-4e43-960a-24c42e43a1ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 13:57:47.540601 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:47.540571 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bae6be5-baf4-4e43-960a-24c42e43a1ad-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5bae6be5-baf4-4e43-960a-24c42e43a1ad" (UID: "5bae6be5-baf4-4e43-960a-24c42e43a1ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:57:47.540696 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:47.540605 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bae6be5-baf4-4e43-960a-24c42e43a1ad-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5bae6be5-baf4-4e43-960a-24c42e43a1ad" (UID: "5bae6be5-baf4-4e43-960a-24c42e43a1ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 13:57:47.540696 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:47.540647 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bae6be5-baf4-4e43-960a-24c42e43a1ad-kube-api-access-vlhjc" (OuterVolumeSpecName: "kube-api-access-vlhjc") pod "5bae6be5-baf4-4e43-960a-24c42e43a1ad" (UID: "5bae6be5-baf4-4e43-960a-24c42e43a1ad"). InnerVolumeSpecName "kube-api-access-vlhjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:57:47.639119 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:47.639085 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bae6be5-baf4-4e43-960a-24c42e43a1ad-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:47.639119 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:47.639117 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bae6be5-baf4-4e43-960a-24c42e43a1ad-metrics-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:47.639119 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:47.639125 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bae6be5-baf4-4e43-960a-24c42e43a1ad-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:47.639329 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:47.639133 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlhjc\" (UniqueName: \"kubernetes.io/projected/5bae6be5-baf4-4e43-960a-24c42e43a1ad-kube-api-access-vlhjc\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:57:47.688758 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:47.688672 2562 generic.go:356] "Generic (PLEG): container finished" podID="5bae6be5-baf4-4e43-960a-24c42e43a1ad" containerID="79a3d3d1f10054985afb9a6c890061e4927165d37ba5b8770e63c417e6a1a1e9" exitCode=0 Sep 16 13:57:47.688758 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:47.688738 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx" Sep 16 13:57:47.688975 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:47.688755 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx" event={"ID":"5bae6be5-baf4-4e43-960a-24c42e43a1ad","Type":"ContainerDied","Data":"79a3d3d1f10054985afb9a6c890061e4927165d37ba5b8770e63c417e6a1a1e9"} Sep 16 13:57:47.688975 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:47.688818 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx" event={"ID":"5bae6be5-baf4-4e43-960a-24c42e43a1ad","Type":"ContainerDied","Data":"f6e5fac6b0faa3ebb7a4febdbcdca8ffe41e7122670708a99b0d98575bea040d"} Sep 16 13:57:47.688975 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:47.688846 2562 scope.go:117] "RemoveContainer" containerID="79a3d3d1f10054985afb9a6c890061e4927165d37ba5b8770e63c417e6a1a1e9" Sep 16 13:57:47.702881 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:47.702647 2562 scope.go:117] "RemoveContainer" containerID="79a3d3d1f10054985afb9a6c890061e4927165d37ba5b8770e63c417e6a1a1e9" Sep 16 13:57:47.703213 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:57:47.703190 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a3d3d1f10054985afb9a6c890061e4927165d37ba5b8770e63c417e6a1a1e9\": container with ID starting with 79a3d3d1f10054985afb9a6c890061e4927165d37ba5b8770e63c417e6a1a1e9 not found: ID does not exist" containerID="79a3d3d1f10054985afb9a6c890061e4927165d37ba5b8770e63c417e6a1a1e9" Sep 16 13:57:47.703277 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:47.703224 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a3d3d1f10054985afb9a6c890061e4927165d37ba5b8770e63c417e6a1a1e9"} err="failed to get container status \"79a3d3d1f10054985afb9a6c890061e4927165d37ba5b8770e63c417e6a1a1e9\": rpc error: code = NotFound desc = could not find container \"79a3d3d1f10054985afb9a6c890061e4927165d37ba5b8770e63c417e6a1a1e9\": container with ID starting with 79a3d3d1f10054985afb9a6c890061e4927165d37ba5b8770e63c417e6a1a1e9 not found: ID does not exist" Sep 16 13:57:47.726018 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:47.725990 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx"] Sep 16 13:57:47.730106 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:47.730085 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f2afd5-5db5cc65ff-pfzkx"] Sep 16 13:57:49.413568 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:57:49.413538 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bae6be5-baf4-4e43-960a-24c42e43a1ad" path="/var/lib/kubelet/pods/5bae6be5-baf4-4e43-960a-24c42e43a1ad/volumes" Sep 16 13:58:07.753158 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753125 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7312-07ad5e"] Sep 16 13:58:07.753650 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753521 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bae6be5-baf4-4e43-960a-24c42e43a1ad" containerName="maas-api" Sep 16 13:58:07.753650 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753533 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bae6be5-baf4-4e43-960a-24c42e43a1ad" containerName="maas-api" Sep 16 13:58:07.753650 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753546 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5427392c-91e4-45ea-bc64-9d9ca1f5b404" containerName="maas-api" Sep 16 13:58:07.753650 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753552 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5427392c-91e4-45ea-bc64-9d9ca1f5b404" containerName="maas-api" Sep 16 13:58:07.753650 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753558 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bbd6dfb-b359-433c-8cc2-3cc44869985b" containerName="maas-api" Sep 16 13:58:07.753650 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753563 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bbd6dfb-b359-433c-8cc2-3cc44869985b" containerName="maas-api" Sep 16 13:58:07.753650 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753576 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aefc8e6-56eb-46a4-b9a5-129a198815b7" containerName="istio-proxy" Sep 16 13:58:07.753650 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753581 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aefc8e6-56eb-46a4-b9a5-129a198815b7" containerName="istio-proxy" Sep 16 13:58:07.753650 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753586 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a22fa4c0-cdee-42dd-ba5d-52ece500282c" containerName="maas-api" Sep 16 13:58:07.753650 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753592 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a22fa4c0-cdee-42dd-ba5d-52ece500282c" containerName="maas-api" Sep 16 13:58:07.753650 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753599 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aefc8e6-56eb-46a4-b9a5-129a198815b7" containerName="istio-proxy" Sep 16 13:58:07.753650 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753604 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aefc8e6-56eb-46a4-b9a5-129a198815b7" containerName="istio-proxy" Sep 16 13:58:07.753650 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753618 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="741e39ff-8c9e-46da-b9b2-6ae7e259825f" containerName="maas-api" Sep 16 13:58:07.753650 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753623 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="741e39ff-8c9e-46da-b9b2-6ae7e259825f" containerName="maas-api" Sep 16 13:58:07.753650 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753631 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df453e94-cc90-4e00-a3f0-7106c99da452" containerName="main" Sep 16 13:58:07.753650 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753636 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="df453e94-cc90-4e00-a3f0-7106c99da452" containerName="main" Sep 16 13:58:07.753650 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753643 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37a19ec4-2357-4035-888e-505953c31e3e" containerName="maas-api" Sep 16 13:58:07.753650 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753648 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="37a19ec4-2357-4035-888e-505953c31e3e" containerName="maas-api" Sep 16 13:58:07.753650 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753654 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13b282d7-4177-478e-b1de-49b9eb1da87b" containerName="main" Sep 16 13:58:07.753650 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753659 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="13b282d7-4177-478e-b1de-49b9eb1da87b" containerName="main" Sep 16 13:58:07.754233 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753712 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="741e39ff-8c9e-46da-b9b2-6ae7e259825f" containerName="maas-api" Sep 16 13:58:07.754233 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753719 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a22fa4c0-cdee-42dd-ba5d-52ece500282c" containerName="maas-api" Sep 16 13:58:07.754233 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753727 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="37a19ec4-2357-4035-888e-505953c31e3e" containerName="maas-api" Sep 16 13:58:07.754233 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753732 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="13b282d7-4177-478e-b1de-49b9eb1da87b" containerName="main" Sep 16 13:58:07.754233 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753738 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bae6be5-baf4-4e43-960a-24c42e43a1ad" containerName="maas-api" Sep 16 13:58:07.754233 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753743 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aefc8e6-56eb-46a4-b9a5-129a198815b7" containerName="istio-proxy" Sep 16 13:58:07.754233 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753748 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5427392c-91e4-45ea-bc64-9d9ca1f5b404" containerName="maas-api" Sep 16 13:58:07.754233 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753753 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aefc8e6-56eb-46a4-b9a5-129a198815b7" containerName="istio-proxy" Sep 16 13:58:07.754233 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753757 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bbd6dfb-b359-433c-8cc2-3cc44869985b" containerName="maas-api" Sep 16 13:58:07.754233 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.753761 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="df453e94-cc90-4e00-a3f0-7106c99da452" containerName="main" Sep 16 13:58:07.758536 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.758520 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7312-07ad5e" Sep 16 13:58:07.762301 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.762282 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2kqls\"" Sep 16 13:58:07.768439 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.768418 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7312-07ad5e"] Sep 16 13:58:07.827267 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.827225 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlplp\" (UniqueName: \"kubernetes.io/projected/d8428da0-0120-494e-8557-4890d96658f5-kube-api-access-mlplp\") pod \"test-curl-7312-07ad5e\" (UID: \"d8428da0-0120-494e-8557-4890d96658f5\") " pod="opendatahub/test-curl-7312-07ad5e" Sep 16 13:58:07.928554 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.928510 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlplp\" (UniqueName: \"kubernetes.io/projected/d8428da0-0120-494e-8557-4890d96658f5-kube-api-access-mlplp\") pod \"test-curl-7312-07ad5e\" (UID: \"d8428da0-0120-494e-8557-4890d96658f5\") " pod="opendatahub/test-curl-7312-07ad5e" Sep 16 13:58:07.939575 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:07.939551 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlplp\" (UniqueName: \"kubernetes.io/projected/d8428da0-0120-494e-8557-4890d96658f5-kube-api-access-mlplp\") pod \"test-curl-7312-07ad5e\" (UID: \"d8428da0-0120-494e-8557-4890d96658f5\") " pod="opendatahub/test-curl-7312-07ad5e" Sep 16 13:58:08.068231 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:08.068135 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7312-07ad5e" Sep 16 13:58:08.368024 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:08.367996 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7312-07ad5e"] Sep 16 13:58:08.369821 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:58:08.369796 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8428da0_0120_494e_8557_4890d96658f5.slice/crio-b436a00f983fb56d56bca1ee5d53c0597a0b94a9c0a49b403e2a51a5f77cd580 WatchSource:0}: Error finding container b436a00f983fb56d56bca1ee5d53c0597a0b94a9c0a49b403e2a51a5f77cd580: Status 404 returned error can't find the container with id b436a00f983fb56d56bca1ee5d53c0597a0b94a9c0a49b403e2a51a5f77cd580 Sep 16 13:58:08.807437 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:08.807403 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7312-07ad5e" event={"ID":"d8428da0-0120-494e-8557-4890d96658f5","Type":"ContainerStarted","Data":"4e76774c2b7bd7a9f2e233e1a49b497345a9a9a45e9218d55a4c13b99ce90ec1"} Sep 16 13:58:08.807437 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:08.807444 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7312-07ad5e" event={"ID":"d8428da0-0120-494e-8557-4890d96658f5","Type":"ContainerStarted","Data":"b436a00f983fb56d56bca1ee5d53c0597a0b94a9c0a49b403e2a51a5f77cd580"} Sep 16 13:58:08.830949 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:08.830906 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7312-07ad5e" podStartSLOduration=1.830891661 podStartE2EDuration="1.830891661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:58:08.830488579 +0000 UTC m=+1649.874791546" watchObservedRunningTime="2026-09-16 13:58:08.830891661 +0000 UTC m=+1649.875194627" Sep 16 13:58:09.626288 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:09.626208 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7312-07ad5e"] Sep 16 13:58:09.630905 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:09.630882 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7312-07ad5e"] Sep 16 13:58:09.812775 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:58:09.812746 2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7312-07ad5e" container="test-curl-7312-07ad5e" Sep 16 13:58:10.817461 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:10.817423 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7312-07ad5e" podUID="d8428da0-0120-494e-8557-4890d96658f5" containerName="test-curl-7312-07ad5e" containerID="cri-o://4e76774c2b7bd7a9f2e233e1a49b497345a9a9a45e9218d55a4c13b99ce90ec1" gracePeriod=2 Sep 16 13:58:10.822695 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:58:10.822669 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7312-07ad5e\" is forbidden: User \"system:node:ip-10-0-139-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-23.ec2.internal' and this object" podUID="d8428da0-0120-494e-8557-4890d96658f5" pod="opendatahub/test-curl-7312-07ad5e" Sep 16 13:58:12.992693 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:12.992666 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7312-07ad5e" Sep 16 13:58:13.076800 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:13.076723 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlplp\" (UniqueName: \"kubernetes.io/projected/d8428da0-0120-494e-8557-4890d96658f5-kube-api-access-mlplp\") pod \"d8428da0-0120-494e-8557-4890d96658f5\" (UID: \"d8428da0-0120-494e-8557-4890d96658f5\") " Sep 16 13:58:13.078900 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:13.078872 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8428da0-0120-494e-8557-4890d96658f5-kube-api-access-mlplp" (OuterVolumeSpecName: "kube-api-access-mlplp") pod "d8428da0-0120-494e-8557-4890d96658f5" (UID: "d8428da0-0120-494e-8557-4890d96658f5"). InnerVolumeSpecName "kube-api-access-mlplp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:58:13.177818 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:13.177771 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlplp\" (UniqueName: \"kubernetes.io/projected/d8428da0-0120-494e-8557-4890d96658f5-kube-api-access-mlplp\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:58:13.414200 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:13.414165 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8428da0-0120-494e-8557-4890d96658f5" path="/var/lib/kubelet/pods/d8428da0-0120-494e-8557-4890d96658f5/volumes" Sep 16 13:58:13.838056 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:13.837967 2562 generic.go:356] "Generic (PLEG): container finished" podID="d8428da0-0120-494e-8557-4890d96658f5" containerID="4e76774c2b7bd7a9f2e233e1a49b497345a9a9a45e9218d55a4c13b99ce90ec1" exitCode=137 Sep 16 13:58:13.838056 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:13.838011 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7312-07ad5e" Sep 16 13:58:13.838287 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:13.838059 2562 scope.go:117] "RemoveContainer" containerID="4e76774c2b7bd7a9f2e233e1a49b497345a9a9a45e9218d55a4c13b99ce90ec1" Sep 16 13:58:13.846894 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:13.846879 2562 scope.go:117] "RemoveContainer" containerID="4e76774c2b7bd7a9f2e233e1a49b497345a9a9a45e9218d55a4c13b99ce90ec1" Sep 16 13:58:13.847158 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:58:13.847138 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e76774c2b7bd7a9f2e233e1a49b497345a9a9a45e9218d55a4c13b99ce90ec1\": container with ID starting with 4e76774c2b7bd7a9f2e233e1a49b497345a9a9a45e9218d55a4c13b99ce90ec1 not found: ID does not exist" containerID="4e76774c2b7bd7a9f2e233e1a49b497345a9a9a45e9218d55a4c13b99ce90ec1" Sep 16 13:58:13.847201 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:58:13.847168 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e76774c2b7bd7a9f2e233e1a49b497345a9a9a45e9218d55a4c13b99ce90ec1"} err="failed to get container status \"4e76774c2b7bd7a9f2e233e1a49b497345a9a9a45e9218d55a4c13b99ce90ec1\": rpc error: code = NotFound desc = could not find container \"4e76774c2b7bd7a9f2e233e1a49b497345a9a9a45e9218d55a4c13b99ce90ec1\": container with ID starting with 4e76774c2b7bd7a9f2e233e1a49b497345a9a9a45e9218d55a4c13b99ce90ec1 not found: ID does not exist" Sep 16 13:59:27.361657 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:27.361620 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7312-565cc9"] Sep 16 13:59:27.362196 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:27.362019 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8428da0-0120-494e-8557-4890d96658f5" containerName="test-curl-7312-07ad5e" Sep 16 13:59:27.362196 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:27.362030 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8428da0-0120-494e-8557-4890d96658f5" containerName="test-curl-7312-07ad5e" Sep 16 13:59:27.362196 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:27.362094 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8428da0-0120-494e-8557-4890d96658f5" containerName="test-curl-7312-07ad5e" Sep 16 13:59:27.365219 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:27.365203 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7312-565cc9" Sep 16 13:59:27.368124 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:27.368106 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2kqls\"" Sep 16 13:59:27.374629 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:27.374609 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7312-565cc9"] Sep 16 13:59:27.472478 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:27.472441 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trhv2\" (UniqueName: \"kubernetes.io/projected/be934e76-75ac-4d8f-8bfc-e2bdcf2f7f04-kube-api-access-trhv2\") pod \"test-curl-7312-565cc9\" (UID: \"be934e76-75ac-4d8f-8bfc-e2bdcf2f7f04\") " pod="opendatahub/test-curl-7312-565cc9" Sep 16 13:59:27.574127 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:27.574079 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trhv2\" (UniqueName: \"kubernetes.io/projected/be934e76-75ac-4d8f-8bfc-e2bdcf2f7f04-kube-api-access-trhv2\") pod \"test-curl-7312-565cc9\" (UID: \"be934e76-75ac-4d8f-8bfc-e2bdcf2f7f04\") " pod="opendatahub/test-curl-7312-565cc9" Sep 16 13:59:27.583443 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:27.583419 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trhv2\" (UniqueName: \"kubernetes.io/projected/be934e76-75ac-4d8f-8bfc-e2bdcf2f7f04-kube-api-access-trhv2\") pod \"test-curl-7312-565cc9\" (UID: \"be934e76-75ac-4d8f-8bfc-e2bdcf2f7f04\") " pod="opendatahub/test-curl-7312-565cc9" Sep 16 13:59:27.675443 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:27.675399 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7312-565cc9" Sep 16 13:59:27.947973 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:27.947943 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7312-565cc9"] Sep 16 13:59:27.949253 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:59:27.949229 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe934e76_75ac_4d8f_8bfc_e2bdcf2f7f04.slice/crio-39b46a10ddb0c3492517464c338881a2c2061d1f3570042bf87aa0909a5fc7a6 WatchSource:0}: Error finding container 39b46a10ddb0c3492517464c338881a2c2061d1f3570042bf87aa0909a5fc7a6: Status 404 returned error can't find the container with id 39b46a10ddb0c3492517464c338881a2c2061d1f3570042bf87aa0909a5fc7a6 Sep 16 13:59:28.256865 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:28.256776 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7312-565cc9" event={"ID":"be934e76-75ac-4d8f-8bfc-e2bdcf2f7f04","Type":"ContainerStarted","Data":"dab95d9e3438f8931c722a81dafdc0c71bc67ad0fc4739f58eaf397456e147be"} Sep 16 13:59:28.256865 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:28.256821 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7312-565cc9" event={"ID":"be934e76-75ac-4d8f-8bfc-e2bdcf2f7f04","Type":"ContainerStarted","Data":"39b46a10ddb0c3492517464c338881a2c2061d1f3570042bf87aa0909a5fc7a6"} Sep 16 13:59:28.274303 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:28.274234 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7312-565cc9" podStartSLOduration=1.274211893 podStartE2EDuration="1.274211893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:59:28.273922766 +0000 UTC m=+1729.318225731" watchObservedRunningTime="2026-09-16 13:59:28.274211893 +0000 UTC m=+1729.318514841" Sep 16 13:59:28.949756 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:28.949668 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7312-565cc9"] Sep 16 13:59:28.954256 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:28.954225 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7312-565cc9"] Sep 16 13:59:29.073161 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:29.073125 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7312-452620"] Sep 16 13:59:29.073614 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:29.073597 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be934e76-75ac-4d8f-8bfc-e2bdcf2f7f04" containerName="test-curl-7312-565cc9" Sep 16 13:59:29.073720 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:29.073616 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="be934e76-75ac-4d8f-8bfc-e2bdcf2f7f04" containerName="test-curl-7312-565cc9" Sep 16 13:59:29.073775 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:29.073739 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="be934e76-75ac-4d8f-8bfc-e2bdcf2f7f04" containerName="test-curl-7312-565cc9" Sep 16 13:59:29.076918 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:29.076900 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7312-452620" Sep 16 13:59:29.082880 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:29.082855 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7312-452620"] Sep 16 13:59:29.189164 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:29.189132 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcrgw\" (UniqueName: \"kubernetes.io/projected/ac8dec74-2284-4da3-9983-d14055136122-kube-api-access-lcrgw\") pod \"test-curl-7312-452620\" (UID: \"ac8dec74-2284-4da3-9983-d14055136122\") " pod="opendatahub/test-curl-7312-452620" Sep 16 13:59:29.262982 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:59:29.262901 2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7312-565cc9" container="test-curl-7312-565cc9" Sep 16 13:59:29.289716 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:29.289693 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcrgw\" (UniqueName: \"kubernetes.io/projected/ac8dec74-2284-4da3-9983-d14055136122-kube-api-access-lcrgw\") pod \"test-curl-7312-452620\" (UID: \"ac8dec74-2284-4da3-9983-d14055136122\") " pod="opendatahub/test-curl-7312-452620" Sep 16 13:59:29.298359 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:29.298328 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcrgw\" (UniqueName: \"kubernetes.io/projected/ac8dec74-2284-4da3-9983-d14055136122-kube-api-access-lcrgw\") pod \"test-curl-7312-452620\" (UID: \"ac8dec74-2284-4da3-9983-d14055136122\") " pod="opendatahub/test-curl-7312-452620" Sep 16 13:59:29.387088 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:29.387057 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7312-452620" Sep 16 13:59:29.660007 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:29.659981 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7312-452620"] Sep 16 13:59:29.661721 ip-10-0-139-23 kubenswrapper[2562]: W0916 13:59:29.661685 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac8dec74_2284_4da3_9983_d14055136122.slice/crio-6eb29995349c1c03ca0907238f6ce4c17ced496a01c15cd24ede2ea9062b4f9b WatchSource:0}: Error finding container 6eb29995349c1c03ca0907238f6ce4c17ced496a01c15cd24ede2ea9062b4f9b: Status 404 returned error can't find the container with id 6eb29995349c1c03ca0907238f6ce4c17ced496a01c15cd24ede2ea9062b4f9b Sep 16 13:59:30.270859 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:30.270825 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7312-452620" event={"ID":"ac8dec74-2284-4da3-9983-d14055136122","Type":"ContainerStarted","Data":"f63b8d99e36831d502ec0b792b37b5a8a73d155e0cc4822e6681b5563a6f49af"} Sep 16 13:59:30.271302 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:30.270868 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7312-452620" event={"ID":"ac8dec74-2284-4da3-9983-d14055136122","Type":"ContainerStarted","Data":"6eb29995349c1c03ca0907238f6ce4c17ced496a01c15cd24ede2ea9062b4f9b"} Sep 16 13:59:30.271302 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:30.270907 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7312-565cc9" podUID="be934e76-75ac-4d8f-8bfc-e2bdcf2f7f04" containerName="test-curl-7312-565cc9" containerID="cri-o://dab95d9e3438f8931c722a81dafdc0c71bc67ad0fc4739f58eaf397456e147be" gracePeriod=2 Sep 16 13:59:30.297558 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:30.297511 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7312-452620" podStartSLOduration=1.297495221 podStartE2EDuration="1.297495221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:59:30.294885739 +0000 UTC m=+1731.339188705" watchObservedRunningTime="2026-09-16 13:59:30.297495221 +0000 UTC m=+1731.341798186" Sep 16 13:59:30.936543 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:30.936507 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7312-452620"] Sep 16 13:59:30.940252 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:30.940225 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7312-452620"] Sep 16 13:59:31.276671 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:59:31.276640 2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7312-452620" container="test-curl-7312-452620" Sep 16 13:59:32.283159 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:32.283123 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7312-452620" podUID="ac8dec74-2284-4da3-9983-d14055136122" containerName="test-curl-7312-452620" containerID="cri-o://f63b8d99e36831d502ec0b792b37b5a8a73d155e0cc4822e6681b5563a6f49af" gracePeriod=2 Sep 16 13:59:32.448596 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:32.448573 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7312-565cc9" Sep 16 13:59:32.517759 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:32.517729 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trhv2\" (UniqueName: \"kubernetes.io/projected/be934e76-75ac-4d8f-8bfc-e2bdcf2f7f04-kube-api-access-trhv2\") pod \"be934e76-75ac-4d8f-8bfc-e2bdcf2f7f04\" (UID: \"be934e76-75ac-4d8f-8bfc-e2bdcf2f7f04\") " Sep 16 13:59:32.519927 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:32.519907 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be934e76-75ac-4d8f-8bfc-e2bdcf2f7f04-kube-api-access-trhv2" (OuterVolumeSpecName: "kube-api-access-trhv2") pod "be934e76-75ac-4d8f-8bfc-e2bdcf2f7f04" (UID: "be934e76-75ac-4d8f-8bfc-e2bdcf2f7f04"). InnerVolumeSpecName "kube-api-access-trhv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:59:32.619146 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:32.619113 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trhv2\" (UniqueName: \"kubernetes.io/projected/be934e76-75ac-4d8f-8bfc-e2bdcf2f7f04-kube-api-access-trhv2\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:59:33.289622 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:33.289585 2562 generic.go:356] "Generic (PLEG): container finished" podID="be934e76-75ac-4d8f-8bfc-e2bdcf2f7f04" containerID="dab95d9e3438f8931c722a81dafdc0c71bc67ad0fc4739f58eaf397456e147be" exitCode=137 Sep 16 13:59:33.290025 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:33.289630 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7312-565cc9" Sep 16 13:59:33.290025 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:33.289686 2562 scope.go:117] "RemoveContainer" containerID="dab95d9e3438f8931c722a81dafdc0c71bc67ad0fc4739f58eaf397456e147be" Sep 16 13:59:33.299339 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:33.299319 2562 scope.go:117] "RemoveContainer" containerID="dab95d9e3438f8931c722a81dafdc0c71bc67ad0fc4739f58eaf397456e147be" Sep 16 13:59:33.299501 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:59:33.299479 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7312-565cc9\" is forbidden: User \"system:node:ip-10-0-139-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-23.ec2.internal' and this object" podUID="be934e76-75ac-4d8f-8bfc-e2bdcf2f7f04" pod="opendatahub/test-curl-7312-565cc9" Sep 16 13:59:33.299676 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:59:33.299654 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dab95d9e3438f8931c722a81dafdc0c71bc67ad0fc4739f58eaf397456e147be\": container with ID starting with dab95d9e3438f8931c722a81dafdc0c71bc67ad0fc4739f58eaf397456e147be not found: ID does not exist" containerID="dab95d9e3438f8931c722a81dafdc0c71bc67ad0fc4739f58eaf397456e147be" Sep 16 13:59:33.299733 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:33.299689 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab95d9e3438f8931c722a81dafdc0c71bc67ad0fc4739f58eaf397456e147be"} err="failed to get container status \"dab95d9e3438f8931c722a81dafdc0c71bc67ad0fc4739f58eaf397456e147be\": rpc error: code = NotFound desc = could not find container \"dab95d9e3438f8931c722a81dafdc0c71bc67ad0fc4739f58eaf397456e147be\": container with ID starting with dab95d9e3438f8931c722a81dafdc0c71bc67ad0fc4739f58eaf397456e147be not found: ID does not exist" Sep 16 13:59:33.413564 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:33.413524 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be934e76-75ac-4d8f-8bfc-e2bdcf2f7f04" path="/var/lib/kubelet/pods/be934e76-75ac-4d8f-8bfc-e2bdcf2f7f04/volumes" Sep 16 13:59:34.472961 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:34.472937 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7312-452620" Sep 16 13:59:34.537107 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:34.537079 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcrgw\" (UniqueName: \"kubernetes.io/projected/ac8dec74-2284-4da3-9983-d14055136122-kube-api-access-lcrgw\") pod \"ac8dec74-2284-4da3-9983-d14055136122\" (UID: \"ac8dec74-2284-4da3-9983-d14055136122\") " Sep 16 13:59:34.539360 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:34.539292 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8dec74-2284-4da3-9983-d14055136122-kube-api-access-lcrgw" (OuterVolumeSpecName: "kube-api-access-lcrgw") pod "ac8dec74-2284-4da3-9983-d14055136122" (UID: "ac8dec74-2284-4da3-9983-d14055136122"). InnerVolumeSpecName "kube-api-access-lcrgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 13:59:34.637772 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:34.637734 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcrgw\" (UniqueName: \"kubernetes.io/projected/ac8dec74-2284-4da3-9983-d14055136122-kube-api-access-lcrgw\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 13:59:35.303630 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:35.303599 2562 generic.go:356] "Generic (PLEG): container finished" podID="ac8dec74-2284-4da3-9983-d14055136122" containerID="f63b8d99e36831d502ec0b792b37b5a8a73d155e0cc4822e6681b5563a6f49af" exitCode=137 Sep 16 13:59:35.303828 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:35.303638 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7312-452620" Sep 16 13:59:35.303828 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:35.303701 2562 scope.go:117] "RemoveContainer" containerID="f63b8d99e36831d502ec0b792b37b5a8a73d155e0cc4822e6681b5563a6f49af" Sep 16 13:59:35.313095 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:35.313077 2562 scope.go:117] "RemoveContainer" containerID="f63b8d99e36831d502ec0b792b37b5a8a73d155e0cc4822e6681b5563a6f49af" Sep 16 13:59:35.313348 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:59:35.313327 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f63b8d99e36831d502ec0b792b37b5a8a73d155e0cc4822e6681b5563a6f49af\": container with ID starting with f63b8d99e36831d502ec0b792b37b5a8a73d155e0cc4822e6681b5563a6f49af not found: ID does not exist" containerID="f63b8d99e36831d502ec0b792b37b5a8a73d155e0cc4822e6681b5563a6f49af" Sep 16 13:59:35.313443 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:35.313355 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f63b8d99e36831d502ec0b792b37b5a8a73d155e0cc4822e6681b5563a6f49af"} err="failed to get container status \"f63b8d99e36831d502ec0b792b37b5a8a73d155e0cc4822e6681b5563a6f49af\": rpc error: code = NotFound desc = could not find container \"f63b8d99e36831d502ec0b792b37b5a8a73d155e0cc4822e6681b5563a6f49af\": container with ID starting with f63b8d99e36831d502ec0b792b37b5a8a73d155e0cc4822e6681b5563a6f49af not found: ID does not exist" Sep 16 13:59:35.314419 ip-10-0-139-23 kubenswrapper[2562]: E0916 13:59:35.314394 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7312-452620\" is forbidden: User \"system:node:ip-10-0-139-23.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-23.ec2.internal' and this object" podUID="ac8dec74-2284-4da3-9983-d14055136122" pod="opendatahub/test-curl-7312-452620" Sep 16 13:59:35.413578 ip-10-0-139-23 kubenswrapper[2562]: I0916 13:59:35.413550 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac8dec74-2284-4da3-9983-d14055136122" path="/var/lib/kubelet/pods/ac8dec74-2284-4da3-9983-d14055136122/volumes" Sep 16 14:00:39.569693 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:00:39.569667 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4xmxv_c7828efb-6be7-4f7b-986f-645a2370163a/console-operator/1.log" Sep 16 14:00:39.576975 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:00:39.576952 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4xmxv_c7828efb-6be7-4f7b-986f-645a2370163a/console-operator/1.log" Sep 16 14:01:00.591884 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:00.591850 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj"] Sep 16 14:01:00.592324 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:00.592232 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac8dec74-2284-4da3-9983-d14055136122" containerName="test-curl-7312-452620" Sep 16 14:01:00.592324 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:00.592244 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac8dec74-2284-4da3-9983-d14055136122" containerName="test-curl-7312-452620" Sep 16 14:01:00.592432 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:00.592336 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac8dec74-2284-4da3-9983-d14055136122" containerName="test-curl-7312-452620" Sep 16 14:01:00.595334 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:00.595318 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj" Sep 16 14:01:00.598391 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:00.598360 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-b59887-serving-cert\"" Sep 16 14:01:00.608111 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:00.608084 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj"] Sep 16 14:01:00.656603 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:00.656568 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d1ad79b-4848-4bb5-8573-641819d3a8b8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj\" (UID: \"9d1ad79b-4848-4bb5-8573-641819d3a8b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj" Sep 16 14:01:00.656768 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:00.656609 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d1ad79b-4848-4bb5-8573-641819d3a8b8-metrics-certs\") pod \"maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj\" (UID: \"9d1ad79b-4848-4bb5-8573-641819d3a8b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj" Sep 16 14:01:00.656768 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:00.656635 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d1ad79b-4848-4bb5-8573-641819d3a8b8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj\" (UID: \"9d1ad79b-4848-4bb5-8573-641819d3a8b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj" Sep 16 14:01:00.656768 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:00.656699 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbvkw\" (UniqueName: \"kubernetes.io/projected/9d1ad79b-4848-4bb5-8573-641819d3a8b8-kube-api-access-cbvkw\") pod \"maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj\" (UID: \"9d1ad79b-4848-4bb5-8573-641819d3a8b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj" Sep 16 14:01:00.757796 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:00.757754 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d1ad79b-4848-4bb5-8573-641819d3a8b8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj\" (UID: \"9d1ad79b-4848-4bb5-8573-641819d3a8b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj" Sep 16 14:01:00.757796 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:00.757802 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d1ad79b-4848-4bb5-8573-641819d3a8b8-metrics-certs\") pod \"maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj\" (UID: \"9d1ad79b-4848-4bb5-8573-641819d3a8b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj" Sep 16 14:01:00.758069 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:00.757820 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d1ad79b-4848-4bb5-8573-641819d3a8b8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj\" (UID: \"9d1ad79b-4848-4bb5-8573-641819d3a8b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj" Sep 16 14:01:00.758069 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:00.757861 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbvkw\" (UniqueName: \"kubernetes.io/projected/9d1ad79b-4848-4bb5-8573-641819d3a8b8-kube-api-access-cbvkw\") pod \"maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj\" (UID: \"9d1ad79b-4848-4bb5-8573-641819d3a8b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj" Sep 16 14:01:00.758530 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:00.758507 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d1ad79b-4848-4bb5-8573-641819d3a8b8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj\" (UID: \"9d1ad79b-4848-4bb5-8573-641819d3a8b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj" Sep 16 14:01:00.760920 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:00.760897 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d1ad79b-4848-4bb5-8573-641819d3a8b8-metrics-certs\") pod \"maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj\" (UID: \"9d1ad79b-4848-4bb5-8573-641819d3a8b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj" Sep 16 14:01:00.761002 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:00.760926 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d1ad79b-4848-4bb5-8573-641819d3a8b8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj\" (UID: \"9d1ad79b-4848-4bb5-8573-641819d3a8b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj" Sep 16 14:01:00.765808 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:00.765785 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbvkw\" (UniqueName: \"kubernetes.io/projected/9d1ad79b-4848-4bb5-8573-641819d3a8b8-kube-api-access-cbvkw\") pod \"maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj\" (UID: \"9d1ad79b-4848-4bb5-8573-641819d3a8b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj" Sep 16 14:01:00.907186 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:00.907159 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj" Sep 16 14:01:01.181573 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:01.181537 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj"] Sep 16 14:01:01.183062 ip-10-0-139-23 kubenswrapper[2562]: W0916 14:01:01.183039 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1ad79b_4848_4bb5_8573_641819d3a8b8.slice/crio-e689ecf97bf26ae29f583c3121f8b56c3df1e2d6fc7bb91e2042e2e5c53e4c76 WatchSource:0}: Error finding container e689ecf97bf26ae29f583c3121f8b56c3df1e2d6fc7bb91e2042e2e5c53e4c76: Status 404 returned error can't find the container with id e689ecf97bf26ae29f583c3121f8b56c3df1e2d6fc7bb91e2042e2e5c53e4c76 Sep 16 14:01:01.184338 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:01.184322 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 14:01:01.788922 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:01.788886 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj" event={"ID":"9d1ad79b-4848-4bb5-8573-641819d3a8b8","Type":"ContainerStarted","Data":"e689ecf97bf26ae29f583c3121f8b56c3df1e2d6fc7bb91e2042e2e5c53e4c76"} Sep 16 14:01:02.795988 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:02.795951 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj" event={"ID":"9d1ad79b-4848-4bb5-8573-641819d3a8b8","Type":"ContainerStarted","Data":"d89671748808dc6e1563fa0f66efb0b833f0904d9f86195d2c2caa88e678d272"} Sep 16 14:01:02.796411 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:02.796040 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj" Sep 16 14:01:02.818092 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:02.818048 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj" podStartSLOduration=1.85166964 podStartE2EDuration="2.818032432s" totalImagesPullingTime="966.362792ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:01:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:01:02.817040163 +0000 UTC m=+1823.861343150" watchObservedRunningTime="2026-09-16 14:01:02.818032432 +0000 UTC m=+1823.862335397" Sep 16 14:01:08.806608 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:08.806580 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj" Sep 16 14:01:30.104628 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:30.104591 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9"] Sep 16 14:01:30.109082 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:30.109062 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9" Sep 16 14:01:30.113299 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:30.113278 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-ac4a3c-serving-cert\"" Sep 16 14:01:30.125954 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:30.125931 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9"] Sep 16 14:01:30.235978 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:30.235945 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ca7313-f774-4687-a226-7fa88139ad7c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9\" (UID: \"09ca7313-f774-4687-a226-7fa88139ad7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9" Sep 16 14:01:30.236157 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:30.236006 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09ca7313-f774-4687-a226-7fa88139ad7c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9\" (UID: \"09ca7313-f774-4687-a226-7fa88139ad7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9" Sep 16 14:01:30.236157 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:30.236025 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09ca7313-f774-4687-a226-7fa88139ad7c-metrics-certs\") pod \"maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9\" (UID: \"09ca7313-f774-4687-a226-7fa88139ad7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9" Sep 16 14:01:30.236157 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:30.236043 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drmjk\" (UniqueName: \"kubernetes.io/projected/09ca7313-f774-4687-a226-7fa88139ad7c-kube-api-access-drmjk\") pod \"maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9\" (UID: \"09ca7313-f774-4687-a226-7fa88139ad7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9" Sep 16 14:01:30.337540 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:30.337507 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09ca7313-f774-4687-a226-7fa88139ad7c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9\" (UID: \"09ca7313-f774-4687-a226-7fa88139ad7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9" Sep 16 14:01:30.337540 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:30.337542 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09ca7313-f774-4687-a226-7fa88139ad7c-metrics-certs\") pod \"maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9\" (UID: \"09ca7313-f774-4687-a226-7fa88139ad7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9" Sep 16 14:01:30.337756 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:30.337561 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drmjk\" (UniqueName: \"kubernetes.io/projected/09ca7313-f774-4687-a226-7fa88139ad7c-kube-api-access-drmjk\") pod \"maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9\" (UID: \"09ca7313-f774-4687-a226-7fa88139ad7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9" Sep 16 14:01:30.337756 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:30.337630 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ca7313-f774-4687-a226-7fa88139ad7c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9\" (UID: \"09ca7313-f774-4687-a226-7fa88139ad7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9" Sep 16 14:01:30.338301 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:30.338271 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ca7313-f774-4687-a226-7fa88139ad7c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9\" (UID: \"09ca7313-f774-4687-a226-7fa88139ad7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9" Sep 16 14:01:30.340408 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:30.340366 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09ca7313-f774-4687-a226-7fa88139ad7c-metrics-certs\") pod \"maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9\" (UID: \"09ca7313-f774-4687-a226-7fa88139ad7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9" Sep 16 14:01:30.340408 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:30.340366 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09ca7313-f774-4687-a226-7fa88139ad7c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9\" (UID: \"09ca7313-f774-4687-a226-7fa88139ad7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9" Sep 16 14:01:30.347934 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:30.347912 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drmjk\" (UniqueName: \"kubernetes.io/projected/09ca7313-f774-4687-a226-7fa88139ad7c-kube-api-access-drmjk\") pod \"maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9\" (UID: \"09ca7313-f774-4687-a226-7fa88139ad7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9" Sep 16 14:01:30.418941 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:30.418895 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9" Sep 16 14:01:30.697136 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:30.697025 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9"] Sep 16 14:01:30.956363 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:30.956278 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9" event={"ID":"09ca7313-f774-4687-a226-7fa88139ad7c","Type":"ContainerStarted","Data":"f4296d1dad7ec2786363759cd75c770390549b0df6566cdc0e29f90c7c88fb32"} Sep 16 14:01:31.963241 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:31.963208 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9" event={"ID":"09ca7313-f774-4687-a226-7fa88139ad7c","Type":"ContainerStarted","Data":"eed8f6e62587b047779f1dccc9d51274074134061e7ce575bde9fea106ad8f44"} Sep 16 14:01:31.963661 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:31.963274 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9" Sep 16 14:01:31.983395 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:31.983317 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9" podStartSLOduration=1.42943598 podStartE2EDuration="1.98329758s" totalImagesPullingTime="553.8616ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:01:31.98247059 +0000 UTC m=+1853.026773600" watchObservedRunningTime="2026-09-16 14:01:31.98329758 +0000 UTC m=+1853.027600548" Sep 16 14:01:37.973589 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:01:37.973556 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9" Sep 16 14:02:56.602546 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:02:56.602511 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh"] Sep 16 14:02:56.606326 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:02:56.606305 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh" Sep 16 14:02:56.609215 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:02:56.609191 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-bf580f-serving-cert\"" Sep 16 14:02:56.618473 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:02:56.618443 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh"] Sep 16 14:02:56.731553 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:02:56.731523 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ebbba09-b7ec-4681-ab68-02922ce0cd5f-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh\" (UID: \"3ebbba09-b7ec-4681-ab68-02922ce0cd5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh" Sep 16 14:02:56.731727 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:02:56.731569 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ebbba09-b7ec-4681-ab68-02922ce0cd5f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh\" (UID: \"3ebbba09-b7ec-4681-ab68-02922ce0cd5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh" Sep 16 14:02:56.731727 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:02:56.731653 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rffxv\" (UniqueName: \"kubernetes.io/projected/3ebbba09-b7ec-4681-ab68-02922ce0cd5f-kube-api-access-rffxv\") pod \"maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh\" (UID: \"3ebbba09-b7ec-4681-ab68-02922ce0cd5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh" Sep 16 14:02:56.731727 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:02:56.731712 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ebbba09-b7ec-4681-ab68-02922ce0cd5f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh\" (UID: \"3ebbba09-b7ec-4681-ab68-02922ce0cd5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh" Sep 16 14:02:56.832934 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:02:56.832905 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ebbba09-b7ec-4681-ab68-02922ce0cd5f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh\" (UID: \"3ebbba09-b7ec-4681-ab68-02922ce0cd5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh" Sep 16 14:02:56.833105 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:02:56.832953 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ebbba09-b7ec-4681-ab68-02922ce0cd5f-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh\" (UID: \"3ebbba09-b7ec-4681-ab68-02922ce0cd5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh" Sep 16 14:02:56.833105 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:02:56.833076 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ebbba09-b7ec-4681-ab68-02922ce0cd5f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh\" (UID: \"3ebbba09-b7ec-4681-ab68-02922ce0cd5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh" Sep 16 14:02:56.833196 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:02:56.833160 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rffxv\" (UniqueName: \"kubernetes.io/projected/3ebbba09-b7ec-4681-ab68-02922ce0cd5f-kube-api-access-rffxv\") pod \"maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh\" (UID: \"3ebbba09-b7ec-4681-ab68-02922ce0cd5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh" Sep 16 14:02:56.833745 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:02:56.833726 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ebbba09-b7ec-4681-ab68-02922ce0cd5f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh\" (UID: \"3ebbba09-b7ec-4681-ab68-02922ce0cd5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh" Sep 16 14:02:56.835654 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:02:56.835608 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ebbba09-b7ec-4681-ab68-02922ce0cd5f-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh\" (UID: \"3ebbba09-b7ec-4681-ab68-02922ce0cd5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh" Sep 16 14:02:56.835654 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:02:56.835643 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ebbba09-b7ec-4681-ab68-02922ce0cd5f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh\" (UID: \"3ebbba09-b7ec-4681-ab68-02922ce0cd5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh" Sep 16 14:02:56.841926 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:02:56.841906 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rffxv\" (UniqueName: \"kubernetes.io/projected/3ebbba09-b7ec-4681-ab68-02922ce0cd5f-kube-api-access-rffxv\") pod \"maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh\" (UID: \"3ebbba09-b7ec-4681-ab68-02922ce0cd5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh" Sep 16 14:02:56.919390 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:02:56.919358 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh" Sep 16 14:02:57.192591 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:02:57.192559 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh"] Sep 16 14:02:57.194178 ip-10-0-139-23 kubenswrapper[2562]: W0916 14:02:57.194138 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ebbba09_b7ec_4681_ab68_02922ce0cd5f.slice/crio-a3c9f4d9d479650fdd46191be645e0782a9be1377a03d1d26d6230e6b67cfa90 WatchSource:0}: Error finding container a3c9f4d9d479650fdd46191be645e0782a9be1377a03d1d26d6230e6b67cfa90: Status 404 returned error can't find the container with id a3c9f4d9d479650fdd46191be645e0782a9be1377a03d1d26d6230e6b67cfa90 Sep 16 14:02:57.442129 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:02:57.442040 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh" event={"ID":"3ebbba09-b7ec-4681-ab68-02922ce0cd5f","Type":"ContainerStarted","Data":"a3c9f4d9d479650fdd46191be645e0782a9be1377a03d1d26d6230e6b67cfa90"} Sep 16 14:02:58.450281 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:02:58.450239 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh" event={"ID":"3ebbba09-b7ec-4681-ab68-02922ce0cd5f","Type":"ContainerStarted","Data":"8c90b7c777ea3666c36f7d1c040aad61b744b5fed03fddaa0f44e698d50ace33"} Sep 16 14:02:58.450760 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:02:58.450552 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh" Sep 16 14:02:58.471764 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:02:58.471718 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh" podStartSLOduration=1.832130926 podStartE2EDuration="2.471702424s" totalImagesPullingTime="639.571498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:02:58.470210412 +0000 UTC m=+1939.514513404" watchObservedRunningTime="2026-09-16 14:02:58.471702424 +0000 UTC m=+1939.516005389" Sep 16 14:03:04.461396 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:04.461347 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh" Sep 16 14:03:34.790132 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:34.790041 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh"] Sep 16 14:03:34.790715 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:34.790331 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh" podUID="3ebbba09-b7ec-4681-ab68-02922ce0cd5f" containerName="maas-api" containerID="cri-o://8c90b7c777ea3666c36f7d1c040aad61b744b5fed03fddaa0f44e698d50ace33" gracePeriod=30 Sep 16 14:03:35.053453 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:35.053361 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh" Sep 16 14:03:35.201470 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:35.201433 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rffxv\" (UniqueName: \"kubernetes.io/projected/3ebbba09-b7ec-4681-ab68-02922ce0cd5f-kube-api-access-rffxv\") pod \"3ebbba09-b7ec-4681-ab68-02922ce0cd5f\" (UID: \"3ebbba09-b7ec-4681-ab68-02922ce0cd5f\") " Sep 16 14:03:35.201664 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:35.201498 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ebbba09-b7ec-4681-ab68-02922ce0cd5f-metrics-certs\") pod \"3ebbba09-b7ec-4681-ab68-02922ce0cd5f\" (UID: \"3ebbba09-b7ec-4681-ab68-02922ce0cd5f\") " Sep 16 14:03:35.201664 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:35.201541 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ebbba09-b7ec-4681-ab68-02922ce0cd5f-openshift-service-ca\") pod \"3ebbba09-b7ec-4681-ab68-02922ce0cd5f\" (UID: \"3ebbba09-b7ec-4681-ab68-02922ce0cd5f\") " Sep 16 14:03:35.201664 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:35.201574 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ebbba09-b7ec-4681-ab68-02922ce0cd5f-maas-api-tls\") pod \"3ebbba09-b7ec-4681-ab68-02922ce0cd5f\" (UID: \"3ebbba09-b7ec-4681-ab68-02922ce0cd5f\") " Sep 16 14:03:35.201948 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:35.201909 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ebbba09-b7ec-4681-ab68-02922ce0cd5f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3ebbba09-b7ec-4681-ab68-02922ce0cd5f" (UID: "3ebbba09-b7ec-4681-ab68-02922ce0cd5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:03:35.204307 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:35.204281 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ebbba09-b7ec-4681-ab68-02922ce0cd5f-kube-api-access-rffxv" (OuterVolumeSpecName: "kube-api-access-rffxv") pod "3ebbba09-b7ec-4681-ab68-02922ce0cd5f" (UID: "3ebbba09-b7ec-4681-ab68-02922ce0cd5f"). InnerVolumeSpecName "kube-api-access-rffxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:03:35.204419 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:35.204342 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ebbba09-b7ec-4681-ab68-02922ce0cd5f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3ebbba09-b7ec-4681-ab68-02922ce0cd5f" (UID: "3ebbba09-b7ec-4681-ab68-02922ce0cd5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:03:35.204419 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:35.204358 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ebbba09-b7ec-4681-ab68-02922ce0cd5f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3ebbba09-b7ec-4681-ab68-02922ce0cd5f" (UID: "3ebbba09-b7ec-4681-ab68-02922ce0cd5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:03:35.302893 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:35.302862 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rffxv\" (UniqueName: \"kubernetes.io/projected/3ebbba09-b7ec-4681-ab68-02922ce0cd5f-kube-api-access-rffxv\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 14:03:35.302893 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:35.302890 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ebbba09-b7ec-4681-ab68-02922ce0cd5f-metrics-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 14:03:35.302893 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:35.302900 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ebbba09-b7ec-4681-ab68-02922ce0cd5f-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 14:03:35.303118 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:35.302912 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ebbba09-b7ec-4681-ab68-02922ce0cd5f-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 14:03:35.661974 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:35.661940 2562 generic.go:356] "Generic (PLEG): container finished" podID="3ebbba09-b7ec-4681-ab68-02922ce0cd5f" containerID="8c90b7c777ea3666c36f7d1c040aad61b744b5fed03fddaa0f44e698d50ace33" exitCode=0 Sep 16 14:03:35.662144 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:35.661983 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh" Sep 16 14:03:35.662144 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:35.662021 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh" event={"ID":"3ebbba09-b7ec-4681-ab68-02922ce0cd5f","Type":"ContainerDied","Data":"8c90b7c777ea3666c36f7d1c040aad61b744b5fed03fddaa0f44e698d50ace33"} Sep 16 14:03:35.662144 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:35.662077 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh" event={"ID":"3ebbba09-b7ec-4681-ab68-02922ce0cd5f","Type":"ContainerDied","Data":"a3c9f4d9d479650fdd46191be645e0782a9be1377a03d1d26d6230e6b67cfa90"} Sep 16 14:03:35.662144 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:35.662106 2562 scope.go:117] "RemoveContainer" containerID="8c90b7c777ea3666c36f7d1c040aad61b744b5fed03fddaa0f44e698d50ace33" Sep 16 14:03:35.674252 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:35.674225 2562 scope.go:117] "RemoveContainer" containerID="8c90b7c777ea3666c36f7d1c040aad61b744b5fed03fddaa0f44e698d50ace33" Sep 16 14:03:35.674580 ip-10-0-139-23 kubenswrapper[2562]: E0916 14:03:35.674548 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c90b7c777ea3666c36f7d1c040aad61b744b5fed03fddaa0f44e698d50ace33\": container with ID starting with 8c90b7c777ea3666c36f7d1c040aad61b744b5fed03fddaa0f44e698d50ace33 not found: ID does not exist" containerID="8c90b7c777ea3666c36f7d1c040aad61b744b5fed03fddaa0f44e698d50ace33" Sep 16 14:03:35.674668 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:35.674587 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c90b7c777ea3666c36f7d1c040aad61b744b5fed03fddaa0f44e698d50ace33"} err="failed to get container status \"8c90b7c777ea3666c36f7d1c040aad61b744b5fed03fddaa0f44e698d50ace33\": rpc error: code = NotFound desc = could not find container \"8c90b7c777ea3666c36f7d1c040aad61b744b5fed03fddaa0f44e698d50ace33\": container with ID starting with 8c90b7c777ea3666c36f7d1c040aad61b744b5fed03fddaa0f44e698d50ace33 not found: ID does not exist" Sep 16 14:03:35.681428 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:35.681401 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh"] Sep 16 14:03:35.687874 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:35.687851 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-bf580f-d4b4f5848-llgrh"] Sep 16 14:03:37.414002 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:03:37.413970 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ebbba09-b7ec-4681-ab68-02922ce0cd5f" path="/var/lib/kubelet/pods/3ebbba09-b7ec-4681-ab68-02922ce0cd5f/volumes" Sep 16 14:04:01.481766 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:01.481726 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj"] Sep 16 14:04:01.482273 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:01.481959 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj" podUID="9d1ad79b-4848-4bb5-8573-641819d3a8b8" containerName="maas-api" containerID="cri-o://d89671748808dc6e1563fa0f66efb0b833f0904d9f86195d2c2caa88e678d272" gracePeriod=30 Sep 16 14:04:01.736895 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:01.736827 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj" Sep 16 14:04:01.813316 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:01.813285 2562 generic.go:356] "Generic (PLEG): container finished" podID="9d1ad79b-4848-4bb5-8573-641819d3a8b8" containerID="d89671748808dc6e1563fa0f66efb0b833f0904d9f86195d2c2caa88e678d272" exitCode=0 Sep 16 14:04:01.813513 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:01.813327 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj" Sep 16 14:04:01.813513 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:01.813371 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj" event={"ID":"9d1ad79b-4848-4bb5-8573-641819d3a8b8","Type":"ContainerDied","Data":"d89671748808dc6e1563fa0f66efb0b833f0904d9f86195d2c2caa88e678d272"} Sep 16 14:04:01.813513 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:01.813446 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj" event={"ID":"9d1ad79b-4848-4bb5-8573-641819d3a8b8","Type":"ContainerDied","Data":"e689ecf97bf26ae29f583c3121f8b56c3df1e2d6fc7bb91e2042e2e5c53e4c76"} Sep 16 14:04:01.813513 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:01.813468 2562 scope.go:117] "RemoveContainer" containerID="d89671748808dc6e1563fa0f66efb0b833f0904d9f86195d2c2caa88e678d272" Sep 16 14:04:01.829008 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:01.828984 2562 scope.go:117] "RemoveContainer" containerID="d89671748808dc6e1563fa0f66efb0b833f0904d9f86195d2c2caa88e678d272" Sep 16 14:04:01.829342 ip-10-0-139-23 kubenswrapper[2562]: E0916 14:04:01.829316 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89671748808dc6e1563fa0f66efb0b833f0904d9f86195d2c2caa88e678d272\": container with ID starting with d89671748808dc6e1563fa0f66efb0b833f0904d9f86195d2c2caa88e678d272 not found: ID does not exist" containerID="d89671748808dc6e1563fa0f66efb0b833f0904d9f86195d2c2caa88e678d272" Sep 16 14:04:01.829427 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:01.829356 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89671748808dc6e1563fa0f66efb0b833f0904d9f86195d2c2caa88e678d272"} err="failed to get container status \"d89671748808dc6e1563fa0f66efb0b833f0904d9f86195d2c2caa88e678d272\": rpc error: code = NotFound desc = could not find container \"d89671748808dc6e1563fa0f66efb0b833f0904d9f86195d2c2caa88e678d272\": container with ID starting with d89671748808dc6e1563fa0f66efb0b833f0904d9f86195d2c2caa88e678d272 not found: ID does not exist" Sep 16 14:04:01.859802 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:01.859777 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d1ad79b-4848-4bb5-8573-641819d3a8b8-openshift-service-ca\") pod \"9d1ad79b-4848-4bb5-8573-641819d3a8b8\" (UID: \"9d1ad79b-4848-4bb5-8573-641819d3a8b8\") " Sep 16 14:04:01.859964 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:01.859838 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d1ad79b-4848-4bb5-8573-641819d3a8b8-maas-api-tls\") pod \"9d1ad79b-4848-4bb5-8573-641819d3a8b8\" (UID: \"9d1ad79b-4848-4bb5-8573-641819d3a8b8\") " Sep 16 14:04:01.859964 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:01.859923 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbvkw\" (UniqueName: \"kubernetes.io/projected/9d1ad79b-4848-4bb5-8573-641819d3a8b8-kube-api-access-cbvkw\") pod \"9d1ad79b-4848-4bb5-8573-641819d3a8b8\" (UID: \"9d1ad79b-4848-4bb5-8573-641819d3a8b8\") " Sep 16 14:04:01.860078 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:01.860015 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d1ad79b-4848-4bb5-8573-641819d3a8b8-metrics-certs\") pod \"9d1ad79b-4848-4bb5-8573-641819d3a8b8\" (UID: \"9d1ad79b-4848-4bb5-8573-641819d3a8b8\") " Sep 16 14:04:01.860136 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:01.860115 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d1ad79b-4848-4bb5-8573-641819d3a8b8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9d1ad79b-4848-4bb5-8573-641819d3a8b8" (UID: "9d1ad79b-4848-4bb5-8573-641819d3a8b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:04:01.860343 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:01.860315 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d1ad79b-4848-4bb5-8573-641819d3a8b8-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 14:04:01.862141 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:01.862114 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d1ad79b-4848-4bb5-8573-641819d3a8b8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9d1ad79b-4848-4bb5-8573-641819d3a8b8" (UID: "9d1ad79b-4848-4bb5-8573-641819d3a8b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:04:01.862245 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:01.862199 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d1ad79b-4848-4bb5-8573-641819d3a8b8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9d1ad79b-4848-4bb5-8573-641819d3a8b8" (UID: "9d1ad79b-4848-4bb5-8573-641819d3a8b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:04:01.862291 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:01.862263 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1ad79b-4848-4bb5-8573-641819d3a8b8-kube-api-access-cbvkw" (OuterVolumeSpecName: "kube-api-access-cbvkw") pod "9d1ad79b-4848-4bb5-8573-641819d3a8b8" (UID: "9d1ad79b-4848-4bb5-8573-641819d3a8b8"). InnerVolumeSpecName "kube-api-access-cbvkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:04:01.961322 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:01.961291 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbvkw\" (UniqueName: \"kubernetes.io/projected/9d1ad79b-4848-4bb5-8573-641819d3a8b8-kube-api-access-cbvkw\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 14:04:01.961322 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:01.961317 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d1ad79b-4848-4bb5-8573-641819d3a8b8-metrics-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 14:04:01.961322 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:01.961325 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d1ad79b-4848-4bb5-8573-641819d3a8b8-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 14:04:02.138150 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:02.138118 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj"] Sep 16 14:04:02.140608 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:02.140590 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b59887-6c6cb6799c-467lj"] Sep 16 14:04:03.418708 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:04:03.418674 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d1ad79b-4848-4bb5-8573-641819d3a8b8" path="/var/lib/kubelet/pods/9d1ad79b-4848-4bb5-8573-641819d3a8b8/volumes" Sep 16 14:05:08.427297 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:08.427213 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9"] Sep 16 14:05:08.427814 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:08.427467 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9" podUID="09ca7313-f774-4687-a226-7fa88139ad7c" containerName="maas-api" containerID="cri-o://eed8f6e62587b047779f1dccc9d51274074134061e7ce575bde9fea106ad8f44" gracePeriod=30 Sep 16 14:05:08.694024 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:08.693990 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9" Sep 16 14:05:08.790207 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:08.790179 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09ca7313-f774-4687-a226-7fa88139ad7c-metrics-certs\") pod \"09ca7313-f774-4687-a226-7fa88139ad7c\" (UID: \"09ca7313-f774-4687-a226-7fa88139ad7c\") " Sep 16 14:05:08.790424 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:08.790265 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ca7313-f774-4687-a226-7fa88139ad7c-openshift-service-ca\") pod \"09ca7313-f774-4687-a226-7fa88139ad7c\" (UID: \"09ca7313-f774-4687-a226-7fa88139ad7c\") " Sep 16 14:05:08.790424 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:08.790310 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09ca7313-f774-4687-a226-7fa88139ad7c-maas-api-tls\") pod \"09ca7313-f774-4687-a226-7fa88139ad7c\" (UID: \"09ca7313-f774-4687-a226-7fa88139ad7c\") " Sep 16 14:05:08.790424 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:08.790347 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drmjk\" (UniqueName: \"kubernetes.io/projected/09ca7313-f774-4687-a226-7fa88139ad7c-kube-api-access-drmjk\") pod \"09ca7313-f774-4687-a226-7fa88139ad7c\" (UID: \"09ca7313-f774-4687-a226-7fa88139ad7c\") " Sep 16 14:05:08.790624 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:08.790595 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ca7313-f774-4687-a226-7fa88139ad7c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "09ca7313-f774-4687-a226-7fa88139ad7c" (UID: "09ca7313-f774-4687-a226-7fa88139ad7c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:05:08.790759 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:08.790740 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ca7313-f774-4687-a226-7fa88139ad7c-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 14:05:08.792599 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:08.792576 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ca7313-f774-4687-a226-7fa88139ad7c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "09ca7313-f774-4687-a226-7fa88139ad7c" (UID: "09ca7313-f774-4687-a226-7fa88139ad7c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:05:08.792599 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:08.792591 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ca7313-f774-4687-a226-7fa88139ad7c-kube-api-access-drmjk" (OuterVolumeSpecName: "kube-api-access-drmjk") pod "09ca7313-f774-4687-a226-7fa88139ad7c" (UID: "09ca7313-f774-4687-a226-7fa88139ad7c"). InnerVolumeSpecName "kube-api-access-drmjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:05:08.792716 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:08.792615 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ca7313-f774-4687-a226-7fa88139ad7c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "09ca7313-f774-4687-a226-7fa88139ad7c" (UID: "09ca7313-f774-4687-a226-7fa88139ad7c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:05:08.892222 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:08.892190 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09ca7313-f774-4687-a226-7fa88139ad7c-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 14:05:08.892222 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:08.892220 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drmjk\" (UniqueName: \"kubernetes.io/projected/09ca7313-f774-4687-a226-7fa88139ad7c-kube-api-access-drmjk\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 14:05:08.892222 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:08.892233 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09ca7313-f774-4687-a226-7fa88139ad7c-metrics-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 14:05:09.214469 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:09.214438 2562 generic.go:356] "Generic (PLEG): container finished" podID="09ca7313-f774-4687-a226-7fa88139ad7c" containerID="eed8f6e62587b047779f1dccc9d51274074134061e7ce575bde9fea106ad8f44" exitCode=0 Sep 16 14:05:09.214469 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:09.214478 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9" event={"ID":"09ca7313-f774-4687-a226-7fa88139ad7c","Type":"ContainerDied","Data":"eed8f6e62587b047779f1dccc9d51274074134061e7ce575bde9fea106ad8f44"} Sep 16 14:05:09.214469 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:09.214481 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9" Sep 16 14:05:09.214773 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:09.214519 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9" event={"ID":"09ca7313-f774-4687-a226-7fa88139ad7c","Type":"ContainerDied","Data":"f4296d1dad7ec2786363759cd75c770390549b0df6566cdc0e29f90c7c88fb32"} Sep 16 14:05:09.214773 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:09.214539 2562 scope.go:117] "RemoveContainer" containerID="eed8f6e62587b047779f1dccc9d51274074134061e7ce575bde9fea106ad8f44" Sep 16 14:05:09.224923 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:09.224889 2562 scope.go:117] "RemoveContainer" containerID="eed8f6e62587b047779f1dccc9d51274074134061e7ce575bde9fea106ad8f44" Sep 16 14:05:09.225223 ip-10-0-139-23 kubenswrapper[2562]: E0916 14:05:09.225191 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed8f6e62587b047779f1dccc9d51274074134061e7ce575bde9fea106ad8f44\": container with ID starting with eed8f6e62587b047779f1dccc9d51274074134061e7ce575bde9fea106ad8f44 not found: ID does not exist" containerID="eed8f6e62587b047779f1dccc9d51274074134061e7ce575bde9fea106ad8f44" Sep 16 14:05:09.225319 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:09.225218 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed8f6e62587b047779f1dccc9d51274074134061e7ce575bde9fea106ad8f44"} err="failed to get container status \"eed8f6e62587b047779f1dccc9d51274074134061e7ce575bde9fea106ad8f44\": rpc error: code = NotFound desc = could not find container \"eed8f6e62587b047779f1dccc9d51274074134061e7ce575bde9fea106ad8f44\": container with ID starting with eed8f6e62587b047779f1dccc9d51274074134061e7ce575bde9fea106ad8f44 not found: ID does not exist" Sep 16 14:05:09.242671 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:09.242644 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9"] Sep 16 14:05:09.246340 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:09.246321 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ac4a3c-7547bd5bb8-6wvh9"] Sep 16 14:05:09.413809 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:09.413779 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09ca7313-f774-4687-a226-7fa88139ad7c" path="/var/lib/kubelet/pods/09ca7313-f774-4687-a226-7fa88139ad7c/volumes" Sep 16 14:05:26.786052 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:26.786017 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8"] Sep 16 14:05:26.786523 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:26.786283 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8" podUID="6f1682ed-9061-4de4-8ebf-6c98237e1718" containerName="maas-api" containerID="cri-o://34830be5b2f44e124dfeb2e71794d545a82bfce0d244fd0fca9b1ed2b203fcdd" gracePeriod=30 Sep 16 14:05:27.038741 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:27.038674 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8" Sep 16 14:05:27.167869 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:27.167837 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f1682ed-9061-4de4-8ebf-6c98237e1718-maas-api-tls\") pod \"6f1682ed-9061-4de4-8ebf-6c98237e1718\" (UID: \"6f1682ed-9061-4de4-8ebf-6c98237e1718\") " Sep 16 14:05:27.168055 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:27.167945 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nn7xx\" (UniqueName: \"kubernetes.io/projected/6f1682ed-9061-4de4-8ebf-6c98237e1718-kube-api-access-nn7xx\") pod \"6f1682ed-9061-4de4-8ebf-6c98237e1718\" (UID: \"6f1682ed-9061-4de4-8ebf-6c98237e1718\") " Sep 16 14:05:27.168055 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:27.167965 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1682ed-9061-4de4-8ebf-6c98237e1718-openshift-service-ca\") pod \"6f1682ed-9061-4de4-8ebf-6c98237e1718\" (UID: \"6f1682ed-9061-4de4-8ebf-6c98237e1718\") " Sep 16 14:05:27.168055 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:27.167999 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f1682ed-9061-4de4-8ebf-6c98237e1718-metrics-certs\") pod \"6f1682ed-9061-4de4-8ebf-6c98237e1718\" (UID: \"6f1682ed-9061-4de4-8ebf-6c98237e1718\") " Sep 16 14:05:27.168322 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:27.168294 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f1682ed-9061-4de4-8ebf-6c98237e1718-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6f1682ed-9061-4de4-8ebf-6c98237e1718" (UID: "6f1682ed-9061-4de4-8ebf-6c98237e1718"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:05:27.170099 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:27.170077 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f1682ed-9061-4de4-8ebf-6c98237e1718-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6f1682ed-9061-4de4-8ebf-6c98237e1718" (UID: "6f1682ed-9061-4de4-8ebf-6c98237e1718"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:05:27.170195 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:27.170165 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f1682ed-9061-4de4-8ebf-6c98237e1718-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6f1682ed-9061-4de4-8ebf-6c98237e1718" (UID: "6f1682ed-9061-4de4-8ebf-6c98237e1718"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:05:27.170309 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:27.170290 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f1682ed-9061-4de4-8ebf-6c98237e1718-kube-api-access-nn7xx" (OuterVolumeSpecName: "kube-api-access-nn7xx") pod "6f1682ed-9061-4de4-8ebf-6c98237e1718" (UID: "6f1682ed-9061-4de4-8ebf-6c98237e1718"). InnerVolumeSpecName "kube-api-access-nn7xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:05:27.269718 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:27.269688 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f1682ed-9061-4de4-8ebf-6c98237e1718-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 14:05:27.269718 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:27.269714 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn7xx\" (UniqueName: \"kubernetes.io/projected/6f1682ed-9061-4de4-8ebf-6c98237e1718-kube-api-access-nn7xx\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 14:05:27.269718 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:27.269724 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1682ed-9061-4de4-8ebf-6c98237e1718-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 14:05:27.269947 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:27.269733 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f1682ed-9061-4de4-8ebf-6c98237e1718-metrics-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 14:05:27.319363 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:27.319273 2562 generic.go:356] "Generic (PLEG): container finished" podID="6f1682ed-9061-4de4-8ebf-6c98237e1718" containerID="34830be5b2f44e124dfeb2e71794d545a82bfce0d244fd0fca9b1ed2b203fcdd" exitCode=0 Sep 16 14:05:27.319363 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:27.319327 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8" Sep 16 14:05:27.319604 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:27.319367 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8" event={"ID":"6f1682ed-9061-4de4-8ebf-6c98237e1718","Type":"ContainerDied","Data":"34830be5b2f44e124dfeb2e71794d545a82bfce0d244fd0fca9b1ed2b203fcdd"} Sep 16 14:05:27.319604 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:27.319442 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8" event={"ID":"6f1682ed-9061-4de4-8ebf-6c98237e1718","Type":"ContainerDied","Data":"0e0bed40e2b84e6f630a0c0970184d220f3e06d3f228aded12f4a14e4552d48f"} Sep 16 14:05:27.319604 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:27.319466 2562 scope.go:117] "RemoveContainer" containerID="34830be5b2f44e124dfeb2e71794d545a82bfce0d244fd0fca9b1ed2b203fcdd" Sep 16 14:05:27.330466 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:27.330449 2562 scope.go:117] "RemoveContainer" containerID="34830be5b2f44e124dfeb2e71794d545a82bfce0d244fd0fca9b1ed2b203fcdd" Sep 16 14:05:27.330740 ip-10-0-139-23 kubenswrapper[2562]: E0916 14:05:27.330719 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34830be5b2f44e124dfeb2e71794d545a82bfce0d244fd0fca9b1ed2b203fcdd\": container with ID starting with 34830be5b2f44e124dfeb2e71794d545a82bfce0d244fd0fca9b1ed2b203fcdd not found: ID does not exist" containerID="34830be5b2f44e124dfeb2e71794d545a82bfce0d244fd0fca9b1ed2b203fcdd" Sep 16 14:05:27.330835 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:27.330745 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34830be5b2f44e124dfeb2e71794d545a82bfce0d244fd0fca9b1ed2b203fcdd"} err="failed to get container status \"34830be5b2f44e124dfeb2e71794d545a82bfce0d244fd0fca9b1ed2b203fcdd\": rpc error: code = NotFound desc = could not find container \"34830be5b2f44e124dfeb2e71794d545a82bfce0d244fd0fca9b1ed2b203fcdd\": container with ID starting with 34830be5b2f44e124dfeb2e71794d545a82bfce0d244fd0fca9b1ed2b203fcdd not found: ID does not exist" Sep 16 14:05:27.347037 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:27.347011 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8"] Sep 16 14:05:27.355848 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:27.355820 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-973610-688b79d587-lwvs8"] Sep 16 14:05:27.413771 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:27.413740 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f1682ed-9061-4de4-8ebf-6c98237e1718" path="/var/lib/kubelet/pods/6f1682ed-9061-4de4-8ebf-6c98237e1718/volumes" Sep 16 14:05:39.610840 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:39.610805 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4xmxv_c7828efb-6be7-4f7b-986f-645a2370163a/console-operator/1.log" Sep 16 14:05:39.618893 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:39.618872 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4xmxv_c7828efb-6be7-4f7b-986f-645a2370163a/console-operator/1.log" Sep 16 14:05:45.641649 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:45.641619 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5"] Sep 16 14:05:45.642113 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:45.641825 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" podUID="04269b40-58d6-47d2-b577-96a84c90f3bf" containerName="maas-api" containerID="cri-o://8e8a8fc9fd0c6033035e068226f04a3daf4d2c8239ffb0a5d0e1639f13e2bc74" gracePeriod=30 Sep 16 14:05:45.915157 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:45.915132 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" Sep 16 14:05:46.036752 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:46.036723 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04269b40-58d6-47d2-b577-96a84c90f3bf-metrics-certs\") pod \"04269b40-58d6-47d2-b577-96a84c90f3bf\" (UID: \"04269b40-58d6-47d2-b577-96a84c90f3bf\") " Sep 16 14:05:46.036752 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:46.036759 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hzl9\" (UniqueName: \"kubernetes.io/projected/04269b40-58d6-47d2-b577-96a84c90f3bf-kube-api-access-4hzl9\") pod \"04269b40-58d6-47d2-b577-96a84c90f3bf\" (UID: \"04269b40-58d6-47d2-b577-96a84c90f3bf\") " Sep 16 14:05:46.037015 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:46.036831 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04269b40-58d6-47d2-b577-96a84c90f3bf-maas-api-tls\") pod \"04269b40-58d6-47d2-b577-96a84c90f3bf\" (UID: \"04269b40-58d6-47d2-b577-96a84c90f3bf\") " Sep 16 14:05:46.037015 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:46.036862 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04269b40-58d6-47d2-b577-96a84c90f3bf-openshift-service-ca\") pod \"04269b40-58d6-47d2-b577-96a84c90f3bf\" (UID: \"04269b40-58d6-47d2-b577-96a84c90f3bf\") " Sep 16 14:05:46.037275 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:46.037248 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04269b40-58d6-47d2-b577-96a84c90f3bf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "04269b40-58d6-47d2-b577-96a84c90f3bf" (UID: "04269b40-58d6-47d2-b577-96a84c90f3bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:05:46.039189 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:46.039166 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04269b40-58d6-47d2-b577-96a84c90f3bf-kube-api-access-4hzl9" (OuterVolumeSpecName: "kube-api-access-4hzl9") pod "04269b40-58d6-47d2-b577-96a84c90f3bf" (UID: "04269b40-58d6-47d2-b577-96a84c90f3bf"). InnerVolumeSpecName "kube-api-access-4hzl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:05:46.039189 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:46.039176 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04269b40-58d6-47d2-b577-96a84c90f3bf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "04269b40-58d6-47d2-b577-96a84c90f3bf" (UID: "04269b40-58d6-47d2-b577-96a84c90f3bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:05:46.039322 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:46.039208 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04269b40-58d6-47d2-b577-96a84c90f3bf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "04269b40-58d6-47d2-b577-96a84c90f3bf" (UID: "04269b40-58d6-47d2-b577-96a84c90f3bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:05:46.138018 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:46.137985 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04269b40-58d6-47d2-b577-96a84c90f3bf-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 14:05:46.138018 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:46.138013 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04269b40-58d6-47d2-b577-96a84c90f3bf-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 14:05:46.138018 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:46.138022 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04269b40-58d6-47d2-b577-96a84c90f3bf-metrics-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 14:05:46.138277 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:46.138031 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hzl9\" (UniqueName: \"kubernetes.io/projected/04269b40-58d6-47d2-b577-96a84c90f3bf-kube-api-access-4hzl9\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 14:05:46.426200 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:46.426148 2562 generic.go:356] "Generic (PLEG): container finished" podID="04269b40-58d6-47d2-b577-96a84c90f3bf" containerID="8e8a8fc9fd0c6033035e068226f04a3daf4d2c8239ffb0a5d0e1639f13e2bc74" exitCode=0 Sep 16 14:05:46.426200 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:46.426196 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" Sep 16 14:05:46.426524 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:46.426213 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" event={"ID":"04269b40-58d6-47d2-b577-96a84c90f3bf","Type":"ContainerDied","Data":"8e8a8fc9fd0c6033035e068226f04a3daf4d2c8239ffb0a5d0e1639f13e2bc74"} Sep 16 14:05:46.426524 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:46.426264 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5" event={"ID":"04269b40-58d6-47d2-b577-96a84c90f3bf","Type":"ContainerDied","Data":"045d2f9b192013025c3f64ad6ba769f0b26281e19212a1a9675579e3c3c00405"} Sep 16 14:05:46.426524 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:46.426285 2562 scope.go:117] "RemoveContainer" containerID="8e8a8fc9fd0c6033035e068226f04a3daf4d2c8239ffb0a5d0e1639f13e2bc74" Sep 16 14:05:46.436000 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:46.435982 2562 scope.go:117] "RemoveContainer" containerID="8e8a8fc9fd0c6033035e068226f04a3daf4d2c8239ffb0a5d0e1639f13e2bc74" Sep 16 14:05:46.436259 ip-10-0-139-23 kubenswrapper[2562]: E0916 14:05:46.436243 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e8a8fc9fd0c6033035e068226f04a3daf4d2c8239ffb0a5d0e1639f13e2bc74\": container with ID starting with 8e8a8fc9fd0c6033035e068226f04a3daf4d2c8239ffb0a5d0e1639f13e2bc74 not found: ID does not exist" containerID="8e8a8fc9fd0c6033035e068226f04a3daf4d2c8239ffb0a5d0e1639f13e2bc74" Sep 16 14:05:46.436303 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:46.436266 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e8a8fc9fd0c6033035e068226f04a3daf4d2c8239ffb0a5d0e1639f13e2bc74"} err="failed to get container status \"8e8a8fc9fd0c6033035e068226f04a3daf4d2c8239ffb0a5d0e1639f13e2bc74\": rpc error: code = NotFound desc = could not find container \"8e8a8fc9fd0c6033035e068226f04a3daf4d2c8239ffb0a5d0e1639f13e2bc74\": container with ID starting with 8e8a8fc9fd0c6033035e068226f04a3daf4d2c8239ffb0a5d0e1639f13e2bc74 not found: ID does not exist" Sep 16 14:05:46.459925 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:46.459896 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5"] Sep 16 14:05:46.477871 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:46.477841 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8c36a6-6cd574bb49-hrfx5"] Sep 16 14:05:47.414525 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:05:47.414499 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04269b40-58d6-47d2-b577-96a84c90f3bf" path="/var/lib/kubelet/pods/04269b40-58d6-47d2-b577-96a84c90f3bf/volumes" Sep 16 14:10:39.651370 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:10:39.651340 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4xmxv_c7828efb-6be7-4f7b-986f-645a2370163a/console-operator/1.log" Sep 16 14:10:39.659587 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:10:39.659567 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4xmxv_c7828efb-6be7-4f7b-986f-645a2370163a/console-operator/1.log" Sep 16 14:11:20.513644 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.513612 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-85cfb86b8-7g8z7"] Sep 16 14:11:20.514259 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.514168 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04269b40-58d6-47d2-b577-96a84c90f3bf" containerName="maas-api" Sep 16 14:11:20.514259 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.514196 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="04269b40-58d6-47d2-b577-96a84c90f3bf" containerName="maas-api" Sep 16 14:11:20.514259 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.514228 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f1682ed-9061-4de4-8ebf-6c98237e1718" containerName="maas-api" Sep 16 14:11:20.514259 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.514238 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f1682ed-9061-4de4-8ebf-6c98237e1718" containerName="maas-api" Sep 16 14:11:20.514259 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.514253 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d1ad79b-4848-4bb5-8573-641819d3a8b8" containerName="maas-api" Sep 16 14:11:20.514259 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.514261 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d1ad79b-4848-4bb5-8573-641819d3a8b8" containerName="maas-api" Sep 16 14:11:20.514619 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.514272 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ebbba09-b7ec-4681-ab68-02922ce0cd5f" containerName="maas-api" Sep 16 14:11:20.514619 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.514280 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ebbba09-b7ec-4681-ab68-02922ce0cd5f" containerName="maas-api" Sep 16 14:11:20.514619 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.514297 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09ca7313-f774-4687-a226-7fa88139ad7c" containerName="maas-api" Sep 16 14:11:20.514619 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.514305 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="09ca7313-f774-4687-a226-7fa88139ad7c" containerName="maas-api" Sep 16 14:11:20.514619 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.514415 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="09ca7313-f774-4687-a226-7fa88139ad7c" containerName="maas-api" Sep 16 14:11:20.514619 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.514432 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f1682ed-9061-4de4-8ebf-6c98237e1718" containerName="maas-api" Sep 16 14:11:20.514619 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.514442 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d1ad79b-4848-4bb5-8573-641819d3a8b8" containerName="maas-api" Sep 16 14:11:20.514619 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.514453 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="04269b40-58d6-47d2-b577-96a84c90f3bf" containerName="maas-api" Sep 16 14:11:20.514619 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.514464 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ebbba09-b7ec-4681-ab68-02922ce0cd5f" containerName="maas-api" Sep 16 14:11:20.517799 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.517780 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85cfb86b8-7g8z7" Sep 16 14:11:20.531234 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.531209 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cfb86b8-7g8z7"] Sep 16 14:11:20.674631 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.674596 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5-maas-api-tls\") pod \"maas-api-85cfb86b8-7g8z7\" (UID: \"f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5\") " pod="odh-ai-gateway-infra/maas-api-85cfb86b8-7g8z7" Sep 16 14:11:20.674828 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.674651 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5-metrics-certs\") pod \"maas-api-85cfb86b8-7g8z7\" (UID: \"f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5\") " pod="odh-ai-gateway-infra/maas-api-85cfb86b8-7g8z7" Sep 16 14:11:20.674828 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.674677 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5-openshift-service-ca\") pod \"maas-api-85cfb86b8-7g8z7\" (UID: \"f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5\") " pod="odh-ai-gateway-infra/maas-api-85cfb86b8-7g8z7" Sep 16 14:11:20.674828 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.674693 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppgr9\" (UniqueName: \"kubernetes.io/projected/f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5-kube-api-access-ppgr9\") pod \"maas-api-85cfb86b8-7g8z7\" (UID: \"f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5\") " pod="odh-ai-gateway-infra/maas-api-85cfb86b8-7g8z7" Sep 16 14:11:20.775262 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.775184 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5-maas-api-tls\") pod \"maas-api-85cfb86b8-7g8z7\" (UID: \"f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5\") " pod="odh-ai-gateway-infra/maas-api-85cfb86b8-7g8z7" Sep 16 14:11:20.775262 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.775239 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5-metrics-certs\") pod \"maas-api-85cfb86b8-7g8z7\" (UID: \"f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5\") " pod="odh-ai-gateway-infra/maas-api-85cfb86b8-7g8z7" Sep 16 14:11:20.775526 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.775279 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5-openshift-service-ca\") pod \"maas-api-85cfb86b8-7g8z7\" (UID: \"f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5\") " pod="odh-ai-gateway-infra/maas-api-85cfb86b8-7g8z7" Sep 16 14:11:20.775526 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.775304 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppgr9\" (UniqueName: \"kubernetes.io/projected/f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5-kube-api-access-ppgr9\") pod \"maas-api-85cfb86b8-7g8z7\" (UID: \"f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5\") " pod="odh-ai-gateway-infra/maas-api-85cfb86b8-7g8z7" Sep 16 14:11:20.776151 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.776078 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5-openshift-service-ca\") pod \"maas-api-85cfb86b8-7g8z7\" (UID: \"f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5\") " pod="odh-ai-gateway-infra/maas-api-85cfb86b8-7g8z7" Sep 16 14:11:20.777786 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.777752 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5-maas-api-tls\") pod \"maas-api-85cfb86b8-7g8z7\" (UID: \"f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5\") " pod="odh-ai-gateway-infra/maas-api-85cfb86b8-7g8z7" Sep 16 14:11:20.777940 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.777912 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5-metrics-certs\") pod \"maas-api-85cfb86b8-7g8z7\" (UID: \"f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5\") " pod="odh-ai-gateway-infra/maas-api-85cfb86b8-7g8z7" Sep 16 14:11:20.783600 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.783579 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppgr9\" (UniqueName: \"kubernetes.io/projected/f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5-kube-api-access-ppgr9\") pod \"maas-api-85cfb86b8-7g8z7\" (UID: \"f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5\") " pod="odh-ai-gateway-infra/maas-api-85cfb86b8-7g8z7" Sep 16 14:11:20.828762 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:20.828732 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85cfb86b8-7g8z7" Sep 16 14:11:21.110102 ip-10-0-139-23 kubenswrapper[2562]: W0916 14:11:21.110075 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ee4e06_84eb_4bd1_bbe9_f4f3c7dcfce5.slice/crio-e8aa7b39b86390cd91f836c1667c86201d5fd9bb76c2fab21c4e5e13b021423d WatchSource:0}: Error finding container e8aa7b39b86390cd91f836c1667c86201d5fd9bb76c2fab21c4e5e13b021423d: Status 404 returned error can't find the container with id e8aa7b39b86390cd91f836c1667c86201d5fd9bb76c2fab21c4e5e13b021423d Sep 16 14:11:21.110952 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:21.110934 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cfb86b8-7g8z7"] Sep 16 14:11:21.111731 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:21.111717 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 14:11:21.287682 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:21.287649 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85cfb86b8-7g8z7" event={"ID":"f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5","Type":"ContainerStarted","Data":"e8aa7b39b86390cd91f836c1667c86201d5fd9bb76c2fab21c4e5e13b021423d"} Sep 16 14:11:22.297739 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:22.297707 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85cfb86b8-7g8z7" event={"ID":"f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5","Type":"ContainerStarted","Data":"1c0910edeefad35723bdf93a123e125ea5ec1897542a66d39808d05b007e6937"} Sep 16 14:11:22.298105 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:22.297820 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-85cfb86b8-7g8z7" Sep 16 14:11:22.324763 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:22.324719 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-85cfb86b8-7g8z7" podStartSLOduration=1.810198022 podStartE2EDuration="2.324699724s" totalImagesPullingTime="514.501702ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:11:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:11:22.322476299 +0000 UTC m=+2443.366779285" watchObservedRunningTime="2026-09-16 14:11:22.324699724 +0000 UTC m=+2443.369002690" Sep 16 14:11:28.308832 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:28.308797 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-85cfb86b8-7g8z7" Sep 16 14:11:38.476773 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:38.476740 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s"] Sep 16 14:11:38.477224 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:38.476965 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s" podUID="7f0f89b7-975b-400c-be50-27ccfd55302f" containerName="maas-api" containerID="cri-o://acf821133e4f6c055619c670a36df2a4115b5ff6dbf94b9c31d31281fafe08c3" gracePeriod=30 Sep 16 14:11:39.716615 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:39.716593 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s" Sep 16 14:11:39.757786 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:39.757755 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkzx7\" (UniqueName: \"kubernetes.io/projected/7f0f89b7-975b-400c-be50-27ccfd55302f-kube-api-access-qkzx7\") pod \"7f0f89b7-975b-400c-be50-27ccfd55302f\" (UID: \"7f0f89b7-975b-400c-be50-27ccfd55302f\") " Sep 16 14:11:39.757964 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:39.757821 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f0f89b7-975b-400c-be50-27ccfd55302f-metrics-certs\") pod \"7f0f89b7-975b-400c-be50-27ccfd55302f\" (UID: \"7f0f89b7-975b-400c-be50-27ccfd55302f\") " Sep 16 14:11:39.757964 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:39.757840 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f0f89b7-975b-400c-be50-27ccfd55302f-maas-api-tls\") pod \"7f0f89b7-975b-400c-be50-27ccfd55302f\" (UID: \"7f0f89b7-975b-400c-be50-27ccfd55302f\") " Sep 16 14:11:39.757964 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:39.757887 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0f89b7-975b-400c-be50-27ccfd55302f-openshift-service-ca\") pod \"7f0f89b7-975b-400c-be50-27ccfd55302f\" (UID: \"7f0f89b7-975b-400c-be50-27ccfd55302f\") " Sep 16 14:11:39.758312 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:39.758280 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0f89b7-975b-400c-be50-27ccfd55302f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7f0f89b7-975b-400c-be50-27ccfd55302f" (UID: "7f0f89b7-975b-400c-be50-27ccfd55302f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 14:11:39.759962 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:39.759934 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0f89b7-975b-400c-be50-27ccfd55302f-kube-api-access-qkzx7" (OuterVolumeSpecName: "kube-api-access-qkzx7") pod "7f0f89b7-975b-400c-be50-27ccfd55302f" (UID: "7f0f89b7-975b-400c-be50-27ccfd55302f"). InnerVolumeSpecName "kube-api-access-qkzx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 14:11:39.760528 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:39.760500 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f0f89b7-975b-400c-be50-27ccfd55302f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7f0f89b7-975b-400c-be50-27ccfd55302f" (UID: "7f0f89b7-975b-400c-be50-27ccfd55302f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:11:39.760609 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:39.760525 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f0f89b7-975b-400c-be50-27ccfd55302f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7f0f89b7-975b-400c-be50-27ccfd55302f" (UID: "7f0f89b7-975b-400c-be50-27ccfd55302f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 14:11:39.836036 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:39.835960 2562 scope.go:117] "RemoveContainer" containerID="acf821133e4f6c055619c670a36df2a4115b5ff6dbf94b9c31d31281fafe08c3" Sep 16 14:11:39.859211 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:39.859187 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkzx7\" (UniqueName: \"kubernetes.io/projected/7f0f89b7-975b-400c-be50-27ccfd55302f-kube-api-access-qkzx7\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 14:11:39.859295 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:39.859214 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f0f89b7-975b-400c-be50-27ccfd55302f-metrics-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 14:11:39.859295 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:39.859223 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f0f89b7-975b-400c-be50-27ccfd55302f-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 14:11:39.859295 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:39.859250 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0f89b7-975b-400c-be50-27ccfd55302f-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\"" Sep 16 14:11:40.405320 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:40.405276 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s" event={"ID":"7f0f89b7-975b-400c-be50-27ccfd55302f","Type":"ContainerDied","Data":"acf821133e4f6c055619c670a36df2a4115b5ff6dbf94b9c31d31281fafe08c3"} Sep 16 14:11:40.405520 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:40.405341 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s" event={"ID":"7f0f89b7-975b-400c-be50-27ccfd55302f","Type":"ContainerDied","Data":"8df7c33afc39c0e785f3aaad322248319116c0ce72bf3a7e42ddeba0cdb83289"} Sep 16 14:11:40.405520 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:40.405295 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s" Sep 16 14:11:40.449098 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:40.449068 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s"] Sep 16 14:11:40.454999 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:40.454970 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-f5f7c95d8-sq75s"] Sep 16 14:11:41.413173 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:41.413133 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f0f89b7-975b-400c-be50-27ccfd55302f" path="/var/lib/kubelet/pods/7f0f89b7-975b-400c-be50-27ccfd55302f/volumes" Sep 16 14:11:50.602921 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:50.602881 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-679b96bc94-5swbb_bbb8d7ff-cded-411c-8b95-1e2eb0751441/authorino/0.log" Sep 16 14:11:56.384783 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:56.384747 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-st8pk_0c0255ad-b36c-497e-a92b-d65807812dd8/manager/0.log" Sep 16 14:11:56.507140 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:56.507114 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8f55z_071707f9-2c28-4395-a424-cdae5211a023/manager/0.log" Sep 16 14:11:57.134251 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:57.134224 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5_24c731c4-8188-4a8b-ae14-34aa43244e9b/copy-manifests/0.log" Sep 16 14:11:57.141235 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:57.141200 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5_24c731c4-8188-4a8b-ae14-34aa43244e9b/manager/0.log" Sep 16 14:11:57.714107 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:57.714078 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-85cfb86b8-7g8z7_f1ee4e06-84eb-4bd1-bbe9-f4f3c7dcfce5/maas-api/0.log" Sep 16 14:11:58.846874 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:58.846845 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-679b96bc94-5swbb_bbb8d7ff-cded-411c-8b95-1e2eb0751441/authorino/0.log" Sep 16 14:11:59.085098 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:59.085053 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-whlj7_75d365c2-8196-4d1d-a77f-fedf7d5b0531/manager/0.log" Sep 16 14:11:59.566580 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:59.566545 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-r2d6q_35a2ca89-4444-401b-a953-1541396b6be0/manager/0.log" Sep 16 14:11:59.915595 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:11:59.915562 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c47gnt_7726ea1c-381a-4673-9401-fe58c2673873/istio-proxy/0.log" Sep 16 14:12:00.392143 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:00.392053 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-tszg8_22fb052b-58e9-441f-8139-f34dcca5ffbb/istio-proxy/1.log" Sep 16 14:12:00.745639 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:00.745606 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-858dfcf476-rkhcs_fbc67ede-7ee2-4ac0-b256-952341815e01/router/0.log" Sep 16 14:12:01.078612 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:01.078533 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-6vg58_a246ae83-c70b-4ca0-9056-39de31224074/main/0.log" Sep 16 14:12:06.824148 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:06.824113 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-skqhh/must-gather-kzht9"] Sep 16 14:12:06.824549 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:06.824518 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f0f89b7-975b-400c-be50-27ccfd55302f" containerName="maas-api" Sep 16 14:12:06.824549 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:06.824530 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f0f89b7-975b-400c-be50-27ccfd55302f" containerName="maas-api" Sep 16 14:12:06.824621 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:06.824609 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f0f89b7-975b-400c-be50-27ccfd55302f" containerName="maas-api" Sep 16 14:12:06.828040 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:06.828024 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-skqhh/must-gather-kzht9" Sep 16 14:12:06.832048 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:06.832027 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-skqhh\"/\"kube-root-ca.crt\"" Sep 16 14:12:06.832048 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:06.832049 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-skqhh\"/\"openshift-service-ca.crt\"" Sep 16 14:12:06.832248 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:06.832031 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-skqhh\"/\"default-dockercfg-x2b6g\"" Sep 16 14:12:06.841169 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:06.841148 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-skqhh/must-gather-kzht9"] Sep 16 14:12:07.015172 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:07.015140 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/eb8b6a4d-dd92-434d-a2bf-fa7d3cdc57d9-must-gather-output\") pod \"must-gather-kzht9\" (UID: \"eb8b6a4d-dd92-434d-a2bf-fa7d3cdc57d9\") " pod="openshift-must-gather-skqhh/must-gather-kzht9" Sep 16 14:12:07.015436 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:07.015179 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndt6p\" (UniqueName: \"kubernetes.io/projected/eb8b6a4d-dd92-434d-a2bf-fa7d3cdc57d9-kube-api-access-ndt6p\") pod \"must-gather-kzht9\" (UID: \"eb8b6a4d-dd92-434d-a2bf-fa7d3cdc57d9\") " pod="openshift-must-gather-skqhh/must-gather-kzht9" Sep 16 14:12:07.116112 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:07.116016 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/eb8b6a4d-dd92-434d-a2bf-fa7d3cdc57d9-must-gather-output\") pod \"must-gather-kzht9\" (UID: \"eb8b6a4d-dd92-434d-a2bf-fa7d3cdc57d9\") " pod="openshift-must-gather-skqhh/must-gather-kzht9" Sep 16 14:12:07.116112 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:07.116066 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndt6p\" (UniqueName: \"kubernetes.io/projected/eb8b6a4d-dd92-434d-a2bf-fa7d3cdc57d9-kube-api-access-ndt6p\") pod \"must-gather-kzht9\" (UID: \"eb8b6a4d-dd92-434d-a2bf-fa7d3cdc57d9\") " pod="openshift-must-gather-skqhh/must-gather-kzht9" Sep 16 14:12:07.116354 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:07.116332 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/eb8b6a4d-dd92-434d-a2bf-fa7d3cdc57d9-must-gather-output\") pod \"must-gather-kzht9\" (UID: \"eb8b6a4d-dd92-434d-a2bf-fa7d3cdc57d9\") " pod="openshift-must-gather-skqhh/must-gather-kzht9" Sep 16 14:12:07.128212 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:07.128184 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndt6p\" (UniqueName: \"kubernetes.io/projected/eb8b6a4d-dd92-434d-a2bf-fa7d3cdc57d9-kube-api-access-ndt6p\") pod \"must-gather-kzht9\" (UID: \"eb8b6a4d-dd92-434d-a2bf-fa7d3cdc57d9\") " pod="openshift-must-gather-skqhh/must-gather-kzht9" Sep 16 14:12:07.137102 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:07.137078 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-skqhh/must-gather-kzht9" Sep 16 14:12:07.413134 ip-10-0-139-23 kubenswrapper[2562]: W0916 14:12:07.413109 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb8b6a4d_dd92_434d_a2bf_fa7d3cdc57d9.slice/crio-d4673af8003aa23cafecd1c1f0d558956741f73598442abf04a3d56efafbb9a9 WatchSource:0}: Error finding container d4673af8003aa23cafecd1c1f0d558956741f73598442abf04a3d56efafbb9a9: Status 404 returned error can't find the container with id d4673af8003aa23cafecd1c1f0d558956741f73598442abf04a3d56efafbb9a9 Sep 16 14:12:07.415738 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:07.415715 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-skqhh/must-gather-kzht9"] Sep 16 14:12:07.553605 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:07.553571 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-skqhh/must-gather-kzht9" event={"ID":"eb8b6a4d-dd92-434d-a2bf-fa7d3cdc57d9","Type":"ContainerStarted","Data":"d4673af8003aa23cafecd1c1f0d558956741f73598442abf04a3d56efafbb9a9"} Sep 16 14:12:10.574692 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:10.574590 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-skqhh/must-gather-kzht9" event={"ID":"eb8b6a4d-dd92-434d-a2bf-fa7d3cdc57d9","Type":"ContainerStarted","Data":"567768ccdb9d1befc9f3c09bbfa4d6d86c12fb9ca5c71bdd06700ee0fcf51484"} Sep 16 14:12:10.574692 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:10.574649 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-skqhh/must-gather-kzht9" event={"ID":"eb8b6a4d-dd92-434d-a2bf-fa7d3cdc57d9","Type":"ContainerStarted","Data":"a63f2d1e6c9f3486985a6a409d34bd26be9098012df33acbe4012250b92d19b3"} Sep 16 14:12:10.594362 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:10.594295 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-skqhh/must-gather-kzht9" podStartSLOduration=1.770600676 podStartE2EDuration="4.594275253s" totalImagesPullingTime="2.823674577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:12:10.591514615 +0000 UTC m=+2491.635817581" watchObservedRunningTime="2026-09-16 14:12:10.594275253 +0000 UTC m=+2491.638578220" Sep 16 14:12:11.782531 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:11.782496 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5s6pv_2a102e1b-9b9b-45f3-9e7e-33794568bdb9/global-pull-secret-syncer/0.log" Sep 16 14:12:11.973216 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:11.973186 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8zxxf_5695f6e6-c46d-4cd0-b4ab-9ab7c4edc9fc/konnectivity-agent/0.log" Sep 16 14:12:12.085622 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:12.085543 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-23.ec2.internal_fc88820cac2d990b85f97c81f96e2d49/haproxy/0.log" Sep 16 14:12:16.162399 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:16.162348 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-679b96bc94-5swbb_bbb8d7ff-cded-411c-8b95-1e2eb0751441/authorino/0.log" Sep 16 14:12:16.221750 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:16.221706 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-whlj7_75d365c2-8196-4d1d-a77f-fedf7d5b0531/manager/0.log" Sep 16 14:12:16.430170 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:16.429681 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-r2d6q_35a2ca89-4444-401b-a953-1541396b6be0/manager/0.log" Sep 16 14:12:18.115035 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:18.115001 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-pw5cd_0d3da9f2-755a-47ea-b78c-2660c79e534c/cluster-monitoring-operator/0.log" Sep 16 14:12:18.214081 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:18.214055 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-84cd89899f-vvm5g_44cef9df-a46b-44ab-8e3e-c3552ebc9174/metrics-server/0.log" Sep 16 14:12:18.247745 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:18.247714 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-t84x2_cab7002e-0e52-4417-84c1-c00e5da634b7/monitoring-plugin/0.log" Sep 16 14:12:18.362271 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:18.362231 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jttjb_624e58c2-824b-4b27-8f19-56f047656011/node-exporter/0.log" Sep 16 14:12:18.384459 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:18.384369 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jttjb_624e58c2-824b-4b27-8f19-56f047656011/kube-rbac-proxy/0.log" Sep 16 14:12:18.405413 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:18.405367 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jttjb_624e58c2-824b-4b27-8f19-56f047656011/init-textfile/0.log" Sep 16 14:12:18.766110 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:18.766076 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-gtb5v_cabf34ef-ee8c-4052-95e9-8f4fe6d550e9/prometheus-operator/0.log" Sep 16 14:12:18.795330 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:18.795298 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-gtb5v_cabf34ef-ee8c-4052-95e9-8f4fe6d550e9/kube-rbac-proxy/0.log" Sep 16 14:12:18.968794 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:18.968637 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d5668779d-z7sst_2dcd994e-d1f5-4da8-b4a7-9627a565cef4/thanos-query/0.log" Sep 16 14:12:18.997153 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:18.997128 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d5668779d-z7sst_2dcd994e-d1f5-4da8-b4a7-9627a565cef4/kube-rbac-proxy-web/0.log" Sep 16 14:12:19.022104 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:19.022026 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d5668779d-z7sst_2dcd994e-d1f5-4da8-b4a7-9627a565cef4/kube-rbac-proxy/0.log" Sep 16 14:12:19.043054 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:19.043022 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d5668779d-z7sst_2dcd994e-d1f5-4da8-b4a7-9627a565cef4/prom-label-proxy/0.log" Sep 16 14:12:19.062768 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:19.062737 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d5668779d-z7sst_2dcd994e-d1f5-4da8-b4a7-9627a565cef4/kube-rbac-proxy-rules/0.log" Sep 16 14:12:19.083063 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:19.083034 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d5668779d-z7sst_2dcd994e-d1f5-4da8-b4a7-9627a565cef4/kube-rbac-proxy-metrics/0.log" Sep 16 14:12:20.526314 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:20.526271 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-6pfrf_186f4dda-b214-4495-b832-c9866c7dabb8/networking-console-plugin/0.log" Sep 16 14:12:21.177335 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:21.177077 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4xmxv_c7828efb-6be7-4f7b-986f-645a2370163a/console-operator/1.log" Sep 16 14:12:21.185321 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:21.185294 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-4xmxv_c7828efb-6be7-4f7b-986f-645a2370163a/console-operator/2.log" Sep 16 14:12:21.808818 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:21.808784 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-ww7v7_e8e0f81e-1e35-49fd-8725-65e1aecac15d/download-server/0.log" Sep 16 14:12:22.446862 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:22.446836 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-q995f_7c11c223-afde-48fd-a3a2-e07f1cee8596/volume-data-source-validator/0.log" Sep 16 14:12:23.385955 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:23.385919 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6"] Sep 16 14:12:23.393268 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:23.393236 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6" Sep 16 14:12:23.400757 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:23.400724 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6"] Sep 16 14:12:23.495066 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:23.495005 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e43f2b8a-9793-4aca-8865-912a7366d3ea-proc\") pod \"perf-node-gather-daemonset-qxqd6\" (UID: \"e43f2b8a-9793-4aca-8865-912a7366d3ea\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6" Sep 16 14:12:23.495066 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:23.495059 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e43f2b8a-9793-4aca-8865-912a7366d3ea-podres\") pod \"perf-node-gather-daemonset-qxqd6\" (UID: \"e43f2b8a-9793-4aca-8865-912a7366d3ea\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6" Sep 16 14:12:23.495327 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:23.495101 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsxqt\" (UniqueName: \"kubernetes.io/projected/e43f2b8a-9793-4aca-8865-912a7366d3ea-kube-api-access-jsxqt\") pod \"perf-node-gather-daemonset-qxqd6\" (UID: \"e43f2b8a-9793-4aca-8865-912a7366d3ea\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6" Sep 16 14:12:23.495327 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:23.495160 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e43f2b8a-9793-4aca-8865-912a7366d3ea-lib-modules\") pod \"perf-node-gather-daemonset-qxqd6\" (UID: \"e43f2b8a-9793-4aca-8865-912a7366d3ea\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6" Sep 16 14:12:23.495327 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:23.495186 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e43f2b8a-9793-4aca-8865-912a7366d3ea-sys\") pod \"perf-node-gather-daemonset-qxqd6\" (UID: \"e43f2b8a-9793-4aca-8865-912a7366d3ea\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6" Sep 16 14:12:23.595873 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:23.595827 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e43f2b8a-9793-4aca-8865-912a7366d3ea-proc\") pod \"perf-node-gather-daemonset-qxqd6\" (UID: \"e43f2b8a-9793-4aca-8865-912a7366d3ea\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6" Sep 16 14:12:23.596057 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:23.595884 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e43f2b8a-9793-4aca-8865-912a7366d3ea-podres\") pod \"perf-node-gather-daemonset-qxqd6\" (UID: \"e43f2b8a-9793-4aca-8865-912a7366d3ea\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6" Sep 16 14:12:23.596057 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:23.595922 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsxqt\" (UniqueName: \"kubernetes.io/projected/e43f2b8a-9793-4aca-8865-912a7366d3ea-kube-api-access-jsxqt\") pod \"perf-node-gather-daemonset-qxqd6\" (UID: \"e43f2b8a-9793-4aca-8865-912a7366d3ea\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6" Sep 16 14:12:23.596057 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:23.595959 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e43f2b8a-9793-4aca-8865-912a7366d3ea-proc\") pod \"perf-node-gather-daemonset-qxqd6\" (UID: \"e43f2b8a-9793-4aca-8865-912a7366d3ea\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6" Sep 16 14:12:23.596057 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:23.595977 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e43f2b8a-9793-4aca-8865-912a7366d3ea-lib-modules\") pod \"perf-node-gather-daemonset-qxqd6\" (UID: \"e43f2b8a-9793-4aca-8865-912a7366d3ea\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6" Sep 16 14:12:23.596057 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:23.596026 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e43f2b8a-9793-4aca-8865-912a7366d3ea-sys\") pod \"perf-node-gather-daemonset-qxqd6\" (UID: \"e43f2b8a-9793-4aca-8865-912a7366d3ea\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6" Sep 16 14:12:23.596420 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:23.596068 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e43f2b8a-9793-4aca-8865-912a7366d3ea-podres\") pod \"perf-node-gather-daemonset-qxqd6\" (UID: \"e43f2b8a-9793-4aca-8865-912a7366d3ea\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6" Sep 16 14:12:23.596420 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:23.596107 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e43f2b8a-9793-4aca-8865-912a7366d3ea-lib-modules\") pod \"perf-node-gather-daemonset-qxqd6\" (UID: \"e43f2b8a-9793-4aca-8865-912a7366d3ea\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6" Sep 16 14:12:23.596420 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:23.596158 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e43f2b8a-9793-4aca-8865-912a7366d3ea-sys\") pod \"perf-node-gather-daemonset-qxqd6\" (UID: \"e43f2b8a-9793-4aca-8865-912a7366d3ea\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6" Sep 16 14:12:23.608883 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:23.607322 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsxqt\" (UniqueName: \"kubernetes.io/projected/e43f2b8a-9793-4aca-8865-912a7366d3ea-kube-api-access-jsxqt\") pod \"perf-node-gather-daemonset-qxqd6\" (UID: \"e43f2b8a-9793-4aca-8865-912a7366d3ea\") " pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6" Sep 16 14:12:23.708681 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:23.708645 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6" Sep 16 14:12:23.798138 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:23.798021 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mxnjp_390ee01e-b850-45d2-b21d-c6b002a1c0ff/dns/0.log" Sep 16 14:12:23.823853 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:23.823798 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mxnjp_390ee01e-b850-45d2-b21d-c6b002a1c0ff/kube-rbac-proxy/0.log" Sep 16 14:12:23.903302 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:23.903273 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t9qw9_24582309-5bb1-4d3d-8a18-a1d7803f287a/dns-node-resolver/0.log" Sep 16 14:12:24.000715 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:24.000682 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6"] Sep 16 14:12:24.001658 ip-10-0-139-23 kubenswrapper[2562]: W0916 14:12:24.001625 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode43f2b8a_9793_4aca_8865_912a7366d3ea.slice/crio-5f75941b5b62f99832782aedba9602fb34f3a8795b1721c460b3228f1b270dff WatchSource:0}: Error finding container 5f75941b5b62f99832782aedba9602fb34f3a8795b1721c460b3228f1b270dff: Status 404 returned error can't find the container with id 5f75941b5b62f99832782aedba9602fb34f3a8795b1721c460b3228f1b270dff Sep 16 14:12:24.505812 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:24.505780 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mcc87_e3cad145-80e4-4662-a1d8-e173c3cddf03/node-ca/0.log" Sep 16 14:12:24.694304 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:24.694274 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6" event={"ID":"e43f2b8a-9793-4aca-8865-912a7366d3ea","Type":"ContainerStarted","Data":"22df07af6d082abfcc1037b59a354b474c26a6c644a6474edf6185691fa51efd"} Sep 16 14:12:24.694504 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:24.694331 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6" Sep 16 14:12:24.694504 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:24.694353 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6" event={"ID":"e43f2b8a-9793-4aca-8865-912a7366d3ea","Type":"ContainerStarted","Data":"5f75941b5b62f99832782aedba9602fb34f3a8795b1721c460b3228f1b270dff"} Sep 16 14:12:24.713478 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:24.713415 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6" podStartSLOduration=1.713344304 podStartE2EDuration="1.713344304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:12:24.711107576 +0000 UTC m=+2505.755410543" watchObservedRunningTime="2026-09-16 14:12:24.713344304 +0000 UTC m=+2505.757647270" Sep 16 14:12:25.550987 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:25.550948 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c47gnt_7726ea1c-381a-4673-9401-fe58c2673873/istio-proxy/0.log" Sep 16 14:12:26.379515 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:26.379481 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-tszg8_22fb052b-58e9-441f-8139-f34dcca5ffbb/istio-proxy/0.log" Sep 16 14:12:26.382486 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:26.382458 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-tszg8_22fb052b-58e9-441f-8139-f34dcca5ffbb/istio-proxy/1.log" Sep 16 14:12:26.489237 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:26.489205 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-858dfcf476-rkhcs_fbc67ede-7ee2-4ac0-b256-952341815e01/router/0.log" Sep 16 14:12:27.412159 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:27.412128 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sfzx5_d5ab9c39-b9a9-42e1-8175-45c414ec2de9/serve-healthcheck-canary/0.log" Sep 16 14:12:27.921478 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:27.921445 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-vdgqh_647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3/insights-operator/0.log" Sep 16 14:12:27.922774 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:27.922753 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-vdgqh_647a9a8e-03c3-421d-9d0c-9dd40dd5dcd3/insights-operator/1.log" Sep 16 14:12:28.014517 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:28.014485 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hvzrx_f940b017-c539-41e8-be1a-c61f0f9ae25d/kube-rbac-proxy/0.log" Sep 16 14:12:28.034876 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:28.034852 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hvzrx_f940b017-c539-41e8-be1a-c61f0f9ae25d/exporter/0.log" Sep 16 14:12:28.056257 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:28.056225 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hvzrx_f940b017-c539-41e8-be1a-c61f0f9ae25d/extractor/0.log" Sep 16 14:12:30.200851 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:30.200805 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-st8pk_0c0255ad-b36c-497e-a92b-d65807812dd8/manager/0.log" Sep 16 14:12:30.715462 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:30.715419 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-skqhh/perf-node-gather-daemonset-qxqd6" Sep 16 14:12:30.728758 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:30.728104 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8f55z_071707f9-2c28-4395-a424-cdae5211a023/manager/0.log" Sep 16 14:12:31.217787 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:31.217758 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5_24c731c4-8188-4a8b-ae14-34aa43244e9b/manager/0.log" Sep 16 14:12:31.239521 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:31.239495 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fdfbf9c59-6rhm5_24c731c4-8188-4a8b-ae14-34aa43244e9b/copy-manifests/0.log" Sep 16 14:12:31.750121 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:31.750086 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c47gnt_7726ea1c-381a-4673-9401-fe58c2673873/istio-proxy/0.log" Sep 16 14:12:32.640738 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:32.640638 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-tszg8_22fb052b-58e9-441f-8139-f34dcca5ffbb/istio-proxy/0.log" Sep 16 14:12:32.642245 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:32.642195 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-tszg8_22fb052b-58e9-441f-8139-f34dcca5ffbb/istio-proxy/1.log" Sep 16 14:12:32.649086 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:32.649062 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8dd669f78-gbkxs_a21e8431-1dd2-4e8b-9acb-fb420faae4f4/manager/0.log" Sep 16 14:12:32.723950 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:32.723888 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-858dfcf476-rkhcs_fbc67ede-7ee2-4ac0-b256-952341815e01/router/0.log" Sep 16 14:12:34.318285 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:34.318258 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c47gnt_7726ea1c-381a-4673-9401-fe58c2673873/istio-proxy/0.log" Sep 16 14:12:35.140057 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:35.140027 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-tszg8_22fb052b-58e9-441f-8139-f34dcca5ffbb/istio-proxy/0.log" Sep 16 14:12:35.145320 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:35.145270 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-tszg8_22fb052b-58e9-441f-8139-f34dcca5ffbb/istio-proxy/1.log" Sep 16 14:12:35.223026 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:35.222986 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-858dfcf476-rkhcs_fbc67ede-7ee2-4ac0-b256-952341815e01/router/0.log" Sep 16 14:12:38.154424 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:38.154372 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-zb868_1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b/kube-storage-version-migrator-operator/1.log" Sep 16 14:12:38.156150 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:38.156127 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-zb868_1e0d4cff-caad-4c86-bbd8-e0b1e86e3d7b/kube-storage-version-migrator-operator/0.log" Sep 16 14:12:39.256539 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:39.256516 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d6bps_870338a6-a8f2-46f8-a5e0-b1b17837057a/kube-multus-additional-cni-plugins/0.log" Sep 16 14:12:39.276656 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:39.276632 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d6bps_870338a6-a8f2-46f8-a5e0-b1b17837057a/egress-router-binary-copy/0.log" Sep 16 14:12:39.295866 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:39.295837 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d6bps_870338a6-a8f2-46f8-a5e0-b1b17837057a/cni-plugins/0.log" Sep 16 14:12:39.316154 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:39.316120 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d6bps_870338a6-a8f2-46f8-a5e0-b1b17837057a/bond-cni-plugin/0.log" Sep 16 14:12:39.336290 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:39.336270 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d6bps_870338a6-a8f2-46f8-a5e0-b1b17837057a/routeoverride-cni/0.log" Sep 16 14:12:39.356477 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:39.356455 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d6bps_870338a6-a8f2-46f8-a5e0-b1b17837057a/whereabouts-cni-bincopy/0.log" Sep 16 14:12:39.375828 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:39.375805 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d6bps_870338a6-a8f2-46f8-a5e0-b1b17837057a/whereabouts-cni/0.log" Sep 16 14:12:39.745126 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:39.745098 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dwt2h_b9f0c124-231d-4677-8696-d7317828eb4f/kube-multus/0.log" Sep 16 14:12:39.924245 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:39.924216 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hm6lg_816a366f-b67b-4679-a323-ab36289dc4b6/network-metrics-daemon/0.log" Sep 16 14:12:39.939520 ip-10-0-139-23 kubenswrapper[2562]: I0916 14:12:39.939496 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hm6lg_816a366f-b67b-4679-a323-ab36289dc4b6/kube-rbac-proxy/0.log"