Sep 11 00:56:05.599629 ip-10-0-135-107 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 11 00:56:05.599641 ip-10-0-135-107 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 11 00:56:05.599648 ip-10-0-135-107 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 11 00:56:05.599812 ip-10-0-135-107 systemd[1]: Failed to start Kubernetes Kubelet. Sep 11 00:56:15.848851 ip-10-0-135-107 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 11 00:56:15.848866 ip-10-0-135-107 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 7b00f10f2f094c1eb17d99f33ba1c4a9 -- Sep 11 00:57:53.060118 ip-10-0-135-107 systemd[1]: Starting Kubernetes Kubelet... Sep 11 00:57:53.482093 ip-10-0-135-107 kubenswrapper[2554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 00:57:53.482093 ip-10-0-135-107 kubenswrapper[2554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 00:57:53.482093 ip-10-0-135-107 kubenswrapper[2554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 00:57:53.482093 ip-10-0-135-107 kubenswrapper[2554]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 00:57:53.482093 ip-10-0-135-107 kubenswrapper[2554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 00:57:53.484676 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.484597 2554 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 00:57:53.488249 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.488234 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 00:57:53.488249 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.488248 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 00:57:53.488325 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.488252 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 00:57:53.488325 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.488256 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 00:57:53.488325 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.488259 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 00:57:53.488447 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.488385 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 00:57:53.488447 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.488407 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 00:57:53.488447 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.488411 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 00:57:53.488447 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.488414 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 00:57:53.488447 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.488418 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 00:57:53.488447 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.488420 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 00:57:53.488447 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.488423 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 00:57:53.488447 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.488426 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 00:57:53.488447 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.488429 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 00:57:53.488447 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.488434 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 00:57:53.489503 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.488438 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 00:57:53.489605 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489596 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 00:57:53.489605 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489604 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 00:57:53.489671 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489608 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 00:57:53.489671 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489611 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 00:57:53.489671 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489614 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 00:57:53.489671 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489617 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 00:57:53.489671 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489620 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 00:57:53.489671 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489623 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 00:57:53.489671 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489626 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 00:57:53.489671 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489631 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 00:57:53.489671 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489641 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 00:57:53.489671 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489646 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 00:57:53.489671 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489649 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 00:57:53.489671 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489653 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 00:57:53.489671 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489657 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 00:57:53.489671 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489660 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 00:57:53.489671 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489663 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 00:57:53.489671 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489666 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 00:57:53.489671 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489668 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 00:57:53.489671 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489671 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 00:57:53.489671 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489674 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 00:57:53.490151 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489677 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 00:57:53.490151 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489680 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 00:57:53.490151 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489684 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 00:57:53.490151 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489687 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 00:57:53.490151 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489689 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 00:57:53.490151 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489692 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 00:57:53.490151 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489696 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 00:57:53.490151 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489700 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 00:57:53.490151 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489702 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 00:57:53.490151 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489705 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 00:57:53.490151 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489707 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 00:57:53.490151 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489710 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 00:57:53.490151 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489713 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 00:57:53.490151 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489715 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 00:57:53.490151 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489718 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 00:57:53.490151 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489722 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 00:57:53.490151 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489725 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 00:57:53.490151 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489728 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 00:57:53.490151 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489730 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 00:57:53.490618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489733 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 00:57:53.490618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489736 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 00:57:53.490618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489738 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 00:57:53.490618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489741 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 00:57:53.490618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489744 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 00:57:53.490618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489746 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 00:57:53.490618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489749 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 00:57:53.490618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489752 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 00:57:53.490618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489754 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 00:57:53.490618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489757 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 00:57:53.490618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489761 2554 feature_gate.go:328] unrecognized feature gate: Example Sep 11 00:57:53.490618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489764 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 00:57:53.490618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489766 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 00:57:53.490618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489770 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 00:57:53.490618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489773 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 00:57:53.490618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489776 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 00:57:53.490618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489779 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 00:57:53.490618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489781 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 00:57:53.490618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489784 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 00:57:53.490618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489786 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 00:57:53.491124 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489789 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 00:57:53.491124 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489791 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 00:57:53.491124 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489794 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 00:57:53.491124 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489797 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 00:57:53.491124 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489800 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 00:57:53.491124 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489802 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 00:57:53.491124 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489805 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 00:57:53.491124 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489807 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 00:57:53.491124 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489810 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 00:57:53.491124 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489813 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 00:57:53.491124 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489815 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 00:57:53.491124 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489819 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 00:57:53.491124 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489822 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 00:57:53.491124 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489825 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 00:57:53.491124 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489827 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 00:57:53.491124 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489830 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 00:57:53.491124 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489832 2554 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 00:57:53.491124 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489835 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 00:57:53.491124 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489838 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 00:57:53.491124 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489854 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 00:57:53.491607 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489857 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 00:57:53.491607 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489859 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 00:57:53.491607 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.489862 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 00:57:53.491607 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490604 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 00:57:53.491607 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490611 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 00:57:53.491607 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490615 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 00:57:53.491607 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490619 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 00:57:53.491607 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490623 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 00:57:53.491607 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490626 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 00:57:53.491607 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490629 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 00:57:53.491607 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490632 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 00:57:53.491607 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490635 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 00:57:53.491607 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490637 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 00:57:53.491607 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490640 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 00:57:53.491607 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490642 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 00:57:53.491607 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490645 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 00:57:53.491607 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490647 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 00:57:53.491607 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490650 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 00:57:53.491607 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490653 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 00:57:53.491607 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490655 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 00:57:53.492235 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490658 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 00:57:53.492235 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490661 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 00:57:53.492235 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490663 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 00:57:53.492235 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490666 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 00:57:53.492235 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490668 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 00:57:53.492235 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490672 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 00:57:53.492235 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490675 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 00:57:53.492235 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490677 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 00:57:53.492235 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490680 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 00:57:53.492235 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490682 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 00:57:53.492235 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490684 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 00:57:53.492235 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490688 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 00:57:53.492235 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490691 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 00:57:53.492235 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490694 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 00:57:53.492235 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490697 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 00:57:53.492235 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490699 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 00:57:53.492235 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490702 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 00:57:53.492235 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490705 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 00:57:53.492235 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490708 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 00:57:53.492235 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490711 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 00:57:53.492741 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490714 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 00:57:53.492741 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490716 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 00:57:53.492741 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490719 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 00:57:53.492741 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490721 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 00:57:53.492741 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490724 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 00:57:53.492741 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490727 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 00:57:53.492741 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490729 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 00:57:53.492741 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490732 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 00:57:53.492741 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490735 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 00:57:53.492741 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490737 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 00:57:53.492741 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490741 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 00:57:53.492741 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490743 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 00:57:53.492741 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490746 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 00:57:53.492741 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490750 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 00:57:53.492741 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490753 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 00:57:53.492741 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490757 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 00:57:53.492741 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490759 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 00:57:53.492741 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490762 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 00:57:53.492741 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490766 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 00:57:53.493228 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490769 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 00:57:53.493228 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490772 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 00:57:53.493228 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490775 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 00:57:53.493228 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490779 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 00:57:53.493228 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490782 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 00:57:53.493228 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490785 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 00:57:53.493228 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490788 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 00:57:53.493228 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490790 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 00:57:53.493228 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490793 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 00:57:53.493228 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490796 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 00:57:53.493228 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490799 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 00:57:53.493228 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490803 2554 feature_gate.go:328] unrecognized feature gate: Example Sep 11 00:57:53.493228 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490805 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 00:57:53.493228 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490808 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 00:57:53.493228 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490811 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 00:57:53.493228 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490814 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 00:57:53.493228 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490817 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 00:57:53.493228 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490820 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 00:57:53.493228 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490823 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 00:57:53.493228 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490825 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 00:57:53.493734 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490828 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 00:57:53.493734 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490831 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 00:57:53.493734 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490833 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 00:57:53.493734 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490836 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 00:57:53.493734 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490853 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 00:57:53.493734 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490856 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 00:57:53.493734 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490859 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 00:57:53.493734 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490861 2554 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 00:57:53.493734 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490864 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 00:57:53.493734 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490867 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 00:57:53.493734 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490870 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 00:57:53.493734 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490872 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 00:57:53.493734 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490876 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 00:57:53.493734 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490878 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 00:57:53.493734 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490881 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 00:57:53.493734 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490884 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 00:57:53.493734 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490886 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 00:57:53.493734 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490888 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 00:57:53.493734 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490891 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 00:57:53.493734 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490893 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 00:57:53.494248 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490896 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 00:57:53.494248 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490898 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 00:57:53.494248 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.490902 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 00:57:53.494248 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491258 2554 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 00:57:53.494248 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491263 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 00:57:53.494248 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491266 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 00:57:53.494248 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491268 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 00:57:53.494248 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491271 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 00:57:53.494248 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491274 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 00:57:53.494248 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491277 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 00:57:53.494248 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491280 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 00:57:53.494248 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491284 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 00:57:53.494248 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491286 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 00:57:53.494248 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491289 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 00:57:53.494248 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491291 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 00:57:53.494248 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491295 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 00:57:53.494248 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491299 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 00:57:53.494248 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491302 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 00:57:53.494248 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491305 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 00:57:53.494718 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491307 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 00:57:53.494718 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491310 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 00:57:53.494718 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491313 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 00:57:53.494718 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491316 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 00:57:53.494718 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491318 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 00:57:53.494718 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491321 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 00:57:53.494718 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491324 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 00:57:53.494718 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491326 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 00:57:53.494718 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491329 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 00:57:53.494718 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491331 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 00:57:53.494718 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491334 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 00:57:53.494718 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491336 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 00:57:53.494718 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491339 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 00:57:53.494718 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491342 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 00:57:53.494718 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491345 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 00:57:53.494718 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491348 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 00:57:53.494718 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491350 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 00:57:53.494718 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491353 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 00:57:53.494718 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491356 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 00:57:53.494718 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491358 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 00:57:53.495223 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491361 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 00:57:53.495223 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491363 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 00:57:53.495223 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491366 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 00:57:53.495223 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491369 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 00:57:53.495223 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491371 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 00:57:53.495223 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491374 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 00:57:53.495223 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491376 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 00:57:53.495223 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491379 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 00:57:53.495223 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491382 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 00:57:53.495223 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491384 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 00:57:53.495223 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491386 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 00:57:53.495223 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491389 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 00:57:53.495223 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491391 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 00:57:53.495223 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491394 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 00:57:53.495223 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491397 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 00:57:53.495223 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491399 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 00:57:53.495223 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491402 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 00:57:53.495223 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491404 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 00:57:53.495223 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491407 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 00:57:53.495223 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491409 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 00:57:53.495714 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491412 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 00:57:53.495714 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491416 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 00:57:53.495714 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491419 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 00:57:53.495714 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491422 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 00:57:53.495714 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491425 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 00:57:53.495714 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491428 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 00:57:53.495714 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491430 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 00:57:53.495714 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491433 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 00:57:53.495714 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491436 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 00:57:53.495714 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491439 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 00:57:53.495714 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491441 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 00:57:53.495714 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491444 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 00:57:53.495714 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491447 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 00:57:53.495714 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491450 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 00:57:53.495714 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491452 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 00:57:53.495714 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491455 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 00:57:53.495714 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491457 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 00:57:53.495714 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491460 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 00:57:53.495714 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491462 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 00:57:53.495714 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491465 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 00:57:53.496208 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491467 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 00:57:53.496208 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491470 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 00:57:53.496208 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491472 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 00:57:53.496208 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491475 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 00:57:53.496208 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491477 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 00:57:53.496208 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491481 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 00:57:53.496208 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491484 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 00:57:53.496208 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491487 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 00:57:53.496208 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491489 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 00:57:53.496208 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491492 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 00:57:53.496208 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491494 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 00:57:53.496208 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491497 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 00:57:53.496208 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491500 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 00:57:53.496208 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491502 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 00:57:53.496208 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491505 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 00:57:53.496208 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491507 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 00:57:53.496208 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491510 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 00:57:53.496208 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491512 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 00:57:53.496208 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491515 2554 feature_gate.go:328] unrecognized feature gate: Example Sep 11 00:57:53.496208 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491518 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 00:57:53.496759 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491521 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 00:57:53.496759 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491523 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 00:57:53.496759 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.491526 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 00:57:53.496759 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491597 2554 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 00:57:53.496759 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491632 2554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 00:57:53.496759 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491639 2554 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 00:57:53.496759 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491644 2554 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 00:57:53.496759 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491648 2554 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 00:57:53.496759 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491651 2554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 00:57:53.496759 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491655 2554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 00:57:53.496759 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491659 2554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 00:57:53.496759 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491662 2554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 00:57:53.496759 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491665 2554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 00:57:53.496759 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491668 2554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 00:57:53.496759 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491672 2554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 00:57:53.496759 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491675 2554 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 00:57:53.496759 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491678 2554 flags.go:64] FLAG: --cgroup-root="" Sep 11 00:57:53.496759 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491683 2554 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 00:57:53.496759 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491686 2554 flags.go:64] FLAG: --client-ca-file="" Sep 11 00:57:53.496759 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491689 2554 flags.go:64] FLAG: --cloud-provider="external" Sep 11 00:57:53.496759 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491692 2554 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 00:57:53.496759 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491696 2554 flags.go:64] FLAG: --cluster-domain="" Sep 11 00:57:53.496759 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491699 2554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 00:57:53.496759 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491702 2554 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 00:57:53.497361 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491708 2554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 00:57:53.497361 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491712 2554 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 00:57:53.497361 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491720 2554 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 00:57:53.497361 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491723 2554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 00:57:53.497361 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491726 2554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 00:57:53.497361 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491729 2554 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 00:57:53.497361 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491732 2554 flags.go:64] FLAG: --contention-profiling="false" Sep 11 00:57:53.497361 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491735 2554 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 00:57:53.497361 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491739 2554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 00:57:53.497361 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491742 2554 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 00:57:53.497361 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491744 2554 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 00:57:53.497361 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491748 2554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 00:57:53.497361 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491751 2554 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 00:57:53.497361 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491754 2554 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 00:57:53.497361 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491757 2554 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 00:57:53.497361 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491760 2554 flags.go:64] FLAG: --enable-server="true" Sep 11 00:57:53.497361 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491763 2554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 00:57:53.497361 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491768 2554 flags.go:64] FLAG: --event-burst="100" Sep 11 00:57:53.497361 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491771 2554 flags.go:64] FLAG: --event-qps="50" Sep 11 00:57:53.497361 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491774 2554 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 00:57:53.497361 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491777 2554 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 00:57:53.497361 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491780 2554 flags.go:64] FLAG: --eviction-hard="" Sep 11 00:57:53.497361 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491783 2554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 00:57:53.497361 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491786 2554 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 00:57:53.497962 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491789 2554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 00:57:53.497962 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491793 2554 flags.go:64] FLAG: --eviction-soft="" Sep 11 00:57:53.497962 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491796 2554 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 00:57:53.497962 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491799 2554 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 00:57:53.497962 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491802 2554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 00:57:53.497962 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491805 2554 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 00:57:53.497962 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491808 2554 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 00:57:53.497962 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491811 2554 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 00:57:53.497962 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491814 2554 flags.go:64] FLAG: --feature-gates="" Sep 11 00:57:53.497962 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491818 2554 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 00:57:53.497962 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491821 2554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 00:57:53.497962 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491825 2554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 00:57:53.497962 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491828 2554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 00:57:53.497962 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491831 2554 flags.go:64] FLAG: --healthz-port="10248" Sep 11 00:57:53.497962 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491834 2554 flags.go:64] FLAG: --help="false" Sep 11 00:57:53.497962 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491837 2554 flags.go:64] FLAG: --hostname-override="ip-10-0-135-107.ec2.internal" Sep 11 00:57:53.497962 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491860 2554 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 00:57:53.497962 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491863 2554 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 00:57:53.497962 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491866 2554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 00:57:53.497962 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491870 2554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 00:57:53.497962 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491873 2554 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 00:57:53.497962 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491876 2554 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 00:57:53.497962 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491879 2554 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491882 2554 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491884 2554 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491887 2554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491891 2554 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491893 2554 flags.go:64] FLAG: --kube-reserved="" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491896 2554 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491899 2554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491902 2554 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491905 2554 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491908 2554 flags.go:64] FLAG: --lock-file="" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491912 2554 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491915 2554 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491918 2554 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491923 2554 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491928 2554 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491931 2554 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491934 2554 flags.go:64] FLAG: --logging-format="text" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491936 2554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491940 2554 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491944 2554 flags.go:64] FLAG: --manifest-url="" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491946 2554 flags.go:64] FLAG: --manifest-url-header="" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491951 2554 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491954 2554 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491958 2554 flags.go:64] FLAG: --max-pods="110" Sep 11 00:57:53.498547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491961 2554 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 00:57:53.499184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491964 2554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 00:57:53.499184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491967 2554 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 00:57:53.499184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491970 2554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 00:57:53.499184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491973 2554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 00:57:53.499184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491976 2554 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 00:57:53.499184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491979 2554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 00:57:53.499184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491987 2554 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 00:57:53.499184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491990 2554 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 00:57:53.499184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491994 2554 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 00:57:53.499184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.491997 2554 flags.go:64] FLAG: --pod-cidr="" Sep 11 00:57:53.499184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492000 2554 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 00:57:53.499184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492004 2554 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 00:57:53.499184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492007 2554 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 00:57:53.499184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492010 2554 flags.go:64] FLAG: --pods-per-core="0" Sep 11 00:57:53.499184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492013 2554 flags.go:64] FLAG: --port="10250" Sep 11 00:57:53.499184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492016 2554 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 00:57:53.499184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492019 2554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0957b0593fec4e399" Sep 11 00:57:53.499184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492022 2554 flags.go:64] FLAG: --qos-reserved="" Sep 11 00:57:53.499184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492026 2554 flags.go:64] FLAG: --read-only-port="10255" Sep 11 00:57:53.499184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492029 2554 flags.go:64] FLAG: --register-node="true" Sep 11 00:57:53.499184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492032 2554 flags.go:64] FLAG: --register-with-taints="" Sep 11 00:57:53.499184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492042 2554 flags.go:64] FLAG: --registry-burst="10" Sep 11 00:57:53.499184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492045 2554 flags.go:64] FLAG: --registry-qps="5" Sep 11 00:57:53.499184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492047 2554 flags.go:64] FLAG: --reserved-cpus="" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492050 2554 flags.go:64] FLAG: --reserved-memory="" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492054 2554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492057 2554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492060 2554 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492063 2554 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492066 2554 flags.go:64] FLAG: --runonce="false" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492069 2554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492072 2554 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492075 2554 flags.go:64] FLAG: --seccomp-default="false" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492078 2554 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492081 2554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492084 2554 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492087 2554 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492090 2554 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492092 2554 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492095 2554 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492098 2554 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492101 2554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492104 2554 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492107 2554 flags.go:64] FLAG: --system-cgroups="" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492110 2554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492115 2554 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492118 2554 flags.go:64] FLAG: --tls-cert-file="" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492120 2554 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 00:57:53.499772 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492124 2554 flags.go:64] FLAG: --tls-min-version="" Sep 11 00:57:53.500426 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492127 2554 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 00:57:53.500426 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492131 2554 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 00:57:53.500426 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492134 2554 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 00:57:53.500426 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492137 2554 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 00:57:53.500426 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492141 2554 flags.go:64] FLAG: --v="2" Sep 11 00:57:53.500426 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492145 2554 flags.go:64] FLAG: --version="false" Sep 11 00:57:53.500426 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492149 2554 flags.go:64] FLAG: --vmodule="" Sep 11 00:57:53.500426 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492153 2554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 00:57:53.500426 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.492156 2554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 00:57:53.500426 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492282 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 00:57:53.500426 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492287 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 00:57:53.500426 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492291 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 00:57:53.500426 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492295 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 00:57:53.500426 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492298 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 00:57:53.500426 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492302 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 00:57:53.500426 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492305 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 00:57:53.500426 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492307 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 00:57:53.500426 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492310 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 00:57:53.500426 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492313 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 00:57:53.500426 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492316 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 00:57:53.500426 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492318 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 00:57:53.500953 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492321 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 00:57:53.500953 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492324 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 00:57:53.500953 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492326 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 00:57:53.500953 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492329 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 00:57:53.500953 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492331 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 00:57:53.500953 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492334 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 00:57:53.500953 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492337 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 00:57:53.500953 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492339 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 00:57:53.500953 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492342 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 00:57:53.500953 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492345 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 00:57:53.500953 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492347 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 00:57:53.500953 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492350 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 00:57:53.500953 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492353 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 00:57:53.500953 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492356 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 00:57:53.500953 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492359 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 00:57:53.500953 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492363 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 00:57:53.500953 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492365 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 00:57:53.500953 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492368 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 00:57:53.500953 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492371 2554 feature_gate.go:328] unrecognized feature gate: Example Sep 11 00:57:53.500953 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492373 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 00:57:53.501449 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492376 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 00:57:53.501449 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492378 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 00:57:53.501449 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492381 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 00:57:53.501449 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492384 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 00:57:53.501449 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492387 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 00:57:53.501449 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492389 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 00:57:53.501449 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492392 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 00:57:53.501449 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492395 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 00:57:53.501449 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492397 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 00:57:53.501449 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492400 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 00:57:53.501449 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492402 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 00:57:53.501449 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492405 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 00:57:53.501449 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492407 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 00:57:53.501449 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492410 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 00:57:53.501449 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492413 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 00:57:53.501449 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492415 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 00:57:53.501449 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492418 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 00:57:53.501449 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492421 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 00:57:53.501449 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492423 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 00:57:53.501449 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492426 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 00:57:53.501958 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492428 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 00:57:53.501958 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492431 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 00:57:53.501958 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492433 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 00:57:53.501958 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492436 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 00:57:53.501958 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492440 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 00:57:53.501958 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492442 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 00:57:53.501958 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492445 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 00:57:53.501958 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492449 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 00:57:53.501958 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492452 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 00:57:53.501958 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492455 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 00:57:53.501958 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492457 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 00:57:53.501958 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492460 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 00:57:53.501958 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492463 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 00:57:53.501958 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492465 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 00:57:53.501958 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492468 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 00:57:53.501958 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492470 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 00:57:53.501958 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492473 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 00:57:53.501958 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492476 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 00:57:53.501958 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492479 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 00:57:53.501958 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492481 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 00:57:53.502462 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492484 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 00:57:53.502462 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492486 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 00:57:53.502462 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492489 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 00:57:53.502462 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492492 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 00:57:53.502462 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492494 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 00:57:53.502462 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492497 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 00:57:53.502462 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492499 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 00:57:53.502462 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492502 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 00:57:53.502462 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492505 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 00:57:53.502462 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492507 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 00:57:53.502462 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492511 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 00:57:53.502462 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492515 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 00:57:53.502462 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492518 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 00:57:53.502462 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492522 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 00:57:53.502462 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492525 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 00:57:53.502462 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492528 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 00:57:53.502462 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492532 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 00:57:53.502462 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492535 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 00:57:53.502462 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492538 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 00:57:53.502948 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492542 2554 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 00:57:53.502948 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492545 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 00:57:53.502948 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492547 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 00:57:53.502948 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492550 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 00:57:53.502948 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492553 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 00:57:53.502948 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492555 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 00:57:53.502948 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492558 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 00:57:53.502948 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.492561 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 00:57:53.502948 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.493208 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 00:57:53.502948 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.499833 2554 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 00:57:53.502948 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.499880 2554 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 00:57:53.502948 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.499898 2554 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 00:57:53.502948 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.499904 2554 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 00:57:53.502948 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.499962 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 00:57:53.502948 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.499967 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 00:57:53.503327 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.499971 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 00:57:53.503327 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.499974 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 00:57:53.503327 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.499977 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 00:57:53.503327 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.499980 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 00:57:53.503327 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.499983 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 00:57:53.503327 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.499986 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 00:57:53.503327 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.499989 2554 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 00:57:53.503327 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.499991 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 00:57:53.503327 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.499994 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 00:57:53.503327 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.499997 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 00:57:53.503327 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.499999 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 00:57:53.503327 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500002 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 00:57:53.503327 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500005 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 00:57:53.503327 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500008 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 00:57:53.503327 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500010 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 00:57:53.503327 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500013 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 00:57:53.503327 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500016 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 00:57:53.503327 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500019 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 00:57:53.503327 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500021 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 00:57:53.503327 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500024 2554 feature_gate.go:328] unrecognized feature gate: Example Sep 11 00:57:53.503867 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500027 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 00:57:53.503867 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500029 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 00:57:53.503867 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500032 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 00:57:53.503867 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500034 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 00:57:53.503867 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500037 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 00:57:53.503867 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500040 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 00:57:53.503867 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500042 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 00:57:53.503867 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500051 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 00:57:53.503867 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500054 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 00:57:53.503867 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500057 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 00:57:53.503867 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500059 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 00:57:53.503867 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500062 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 00:57:53.503867 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500064 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 00:57:53.503867 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500067 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 00:57:53.503867 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500070 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 00:57:53.503867 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500072 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 00:57:53.503867 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500075 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 00:57:53.503867 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500077 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 00:57:53.503867 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500080 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 00:57:53.503867 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500083 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 00:57:53.504346 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500085 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 00:57:53.504346 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500088 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 00:57:53.504346 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500090 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 00:57:53.504346 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500093 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 00:57:53.504346 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500096 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 00:57:53.504346 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500099 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 00:57:53.504346 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500102 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 00:57:53.504346 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500104 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 00:57:53.504346 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500107 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 00:57:53.504346 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500109 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 00:57:53.504346 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500112 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 00:57:53.504346 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500115 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 00:57:53.504346 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500119 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 00:57:53.504346 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500122 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 00:57:53.504346 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500126 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 00:57:53.504346 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500129 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 00:57:53.504346 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500131 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 00:57:53.504346 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500134 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 00:57:53.504346 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500137 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 00:57:53.504346 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500140 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 00:57:53.504819 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500143 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 00:57:53.504819 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500146 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 00:57:53.504819 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500149 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 00:57:53.504819 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500152 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 00:57:53.504819 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500154 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 00:57:53.504819 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500157 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 00:57:53.504819 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500161 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 00:57:53.504819 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500165 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 00:57:53.504819 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500169 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 00:57:53.504819 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500173 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 00:57:53.504819 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500176 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 00:57:53.504819 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500179 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 00:57:53.504819 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500182 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 00:57:53.504819 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500185 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 00:57:53.504819 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500187 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 00:57:53.504819 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500190 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 00:57:53.504819 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500193 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 00:57:53.504819 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500196 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 00:57:53.504819 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500198 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 00:57:53.505366 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500201 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 00:57:53.505366 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500203 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 00:57:53.505366 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500206 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 00:57:53.505366 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500208 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 00:57:53.505366 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500211 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 00:57:53.505366 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500214 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 00:57:53.505366 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500217 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 00:57:53.505366 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500219 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 00:57:53.505366 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500222 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 00:57:53.505366 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500225 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 00:57:53.505366 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500227 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 00:57:53.505366 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500230 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 00:57:53.505366 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500232 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 00:57:53.505366 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500235 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 00:57:53.505366 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500238 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 00:57:53.505366 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500241 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 00:57:53.505366 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500244 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 00:57:53.505366 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500246 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 00:57:53.505804 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.500251 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 00:57:53.505804 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500358 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 00:57:53.505804 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500362 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 00:57:53.505804 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500365 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 00:57:53.505804 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500368 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 00:57:53.505804 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500371 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 00:57:53.505804 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500374 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 00:57:53.505804 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500377 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 00:57:53.505804 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500379 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 00:57:53.505804 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500382 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 00:57:53.505804 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500386 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 00:57:53.505804 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500390 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 00:57:53.505804 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500393 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 00:57:53.505804 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500396 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 00:57:53.506185 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500399 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 00:57:53.506185 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500402 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 00:57:53.506185 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500405 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 00:57:53.506185 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500408 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 00:57:53.506185 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500410 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 00:57:53.506185 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500413 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 00:57:53.506185 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500416 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 00:57:53.506185 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500418 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 00:57:53.506185 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500421 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 00:57:53.506185 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500423 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 00:57:53.506185 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500426 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 00:57:53.506185 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500429 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 00:57:53.506185 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500431 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 00:57:53.506185 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500434 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 00:57:53.506185 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500437 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 00:57:53.506185 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500440 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 00:57:53.506185 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500443 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 00:57:53.506185 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500447 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 00:57:53.506185 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500449 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 00:57:53.506649 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500452 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 00:57:53.506649 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500455 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 00:57:53.506649 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500457 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 00:57:53.506649 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500460 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 00:57:53.506649 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500462 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 00:57:53.506649 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500465 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 00:57:53.506649 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500467 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 00:57:53.506649 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500470 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 00:57:53.506649 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500472 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 00:57:53.506649 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500475 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 00:57:53.506649 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500477 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 00:57:53.506649 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500480 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 00:57:53.506649 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500482 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 00:57:53.506649 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500485 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 00:57:53.506649 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500488 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 00:57:53.506649 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500490 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 00:57:53.506649 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500493 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 00:57:53.506649 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500496 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 00:57:53.506649 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500498 2554 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 00:57:53.506649 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500501 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 00:57:53.507145 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500503 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 00:57:53.507145 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500506 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 00:57:53.507145 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500509 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 00:57:53.507145 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500511 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 00:57:53.507145 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500514 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 00:57:53.507145 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500517 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 00:57:53.507145 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500519 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 00:57:53.507145 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500522 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 00:57:53.507145 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500524 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 00:57:53.507145 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500527 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 00:57:53.507145 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500530 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 00:57:53.507145 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500533 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 00:57:53.507145 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500535 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 00:57:53.507145 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500538 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 00:57:53.507145 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500540 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 00:57:53.507145 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500543 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 00:57:53.507145 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500545 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 00:57:53.507145 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500548 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 00:57:53.507145 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500550 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 00:57:53.507145 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500553 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 00:57:53.507618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500555 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 00:57:53.507618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500558 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 00:57:53.507618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500561 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 00:57:53.507618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500564 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 00:57:53.507618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500566 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 00:57:53.507618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500570 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 00:57:53.507618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500574 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 00:57:53.507618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500577 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 00:57:53.507618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500580 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 00:57:53.507618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500582 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 00:57:53.507618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500585 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 00:57:53.507618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500587 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 00:57:53.507618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500590 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 00:57:53.507618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500593 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 00:57:53.507618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500595 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 00:57:53.507618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500597 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 00:57:53.507618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500600 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 00:57:53.507618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500603 2554 feature_gate.go:328] unrecognized feature gate: Example Sep 11 00:57:53.507618 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500605 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 00:57:53.508102 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500608 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 00:57:53.508102 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500610 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 00:57:53.508102 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500613 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 00:57:53.508102 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500616 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 00:57:53.508102 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500619 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 00:57:53.508102 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500622 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 00:57:53.508102 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500624 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 00:57:53.508102 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:53.500627 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 00:57:53.508102 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.500631 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 00:57:53.508102 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.500721 2554 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 00:57:53.508102 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.502596 2554 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 00:57:53.508102 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.503560 2554 server.go:1019] "Starting client certificate rotation" Sep 11 00:57:53.508102 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.503660 2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 00:57:53.508102 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.503692 2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 00:57:53.508505 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.505917 2554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 00:57:53.508505 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.507742 2554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 00:57:53.516983 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.516963 2554 log.go:25] "Validated CRI v1 runtime API" Sep 11 00:57:53.520697 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.520683 2554 log.go:25] "Validated CRI v1 image API" Sep 11 00:57:53.521486 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.521469 2554 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 00:57:53.523968 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.523948 2554 fs.go:135] Filesystem UUIDs: map[53c5875a-841f-4e8e-8871-b813b2663474:/dev/nvme0n1p3 5ddc83e2-4e5f-41dc-9b1b-bb87e69676bb:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Sep 11 00:57:53.524030 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.523969 2554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 00:57:53.530260 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.530152 2554 manager.go:216] Machine: {Timestamp:2026-09-11 00:57:53.528857171 +0000 UTC m=+0.352310672 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098210 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20b2541b94cd9d9108ad805d747b53 SystemUUID:ec20b254-1b94-cd9d-9108-ad805d747b53 BootID:7b00f10f-2f09-4c1e-b17d-99f33ba1c4a9 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:88:cf:21:46:03 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:88:cf:21:46:03 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:c8:c2:3e:24:42 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 00:57:53.530260 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.530253 2554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 00:57:53.530384 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.530323 2554 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 00:57:53.530997 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.530983 2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 00:57:53.533535 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.533510 2554 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 00:57:53.533681 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.533539 2554 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-107.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 00:57:53.534248 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.534237 2554 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 00:57:53.534280 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.534249 2554 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 00:57:53.534280 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.534261 2554 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 00:57:53.534280 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.534274 2554 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 00:57:53.535049 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.535039 2554 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 00:57:53.538604 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.538592 2554 state_mem.go:36] "Initialized new in-memory state store" Sep 11 00:57:53.538718 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.538709 2554 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 00:57:53.538758 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.538749 2554 kubelet.go:486] "Attempting to sync node with API server" Sep 11 00:57:53.538804 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.538774 2554 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 00:57:53.538804 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.538794 2554 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 00:57:53.538889 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.538807 2554 kubelet.go:398] "Adding apiserver pod source" Sep 11 00:57:53.538889 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.538827 2554 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 00:57:53.539230 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.539214 2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 00:57:53.539230 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.539234 2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 00:57:53.541428 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.541413 2554 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 00:57:53.543711 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.543692 2554 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 00:57:53.543762 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.543721 2554 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 00:57:53.545155 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.545143 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 00:57:53.545192 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.545170 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 00:57:53.545192 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.545180 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 00:57:53.545192 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.545189 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 00:57:53.545280 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.545202 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 00:57:53.545280 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.545211 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 00:57:53.545280 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.545241 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 00:57:53.545280 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.545248 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 00:57:53.545280 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.545256 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 00:57:53.545280 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.545261 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 00:57:53.545280 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.545277 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 00:57:53.545280 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.545282 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 00:57:53.545477 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.545314 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 00:57:53.545477 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.545320 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 00:57:53.548872 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.548828 2554 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-107.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 00:57:53.548872 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:53.548834 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 00:57:53.549024 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.548931 2554 server.go:1268] "Started kubelet" Sep 11 00:57:53.549127 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.549101 2554 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 00:57:53.549239 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.549147 2554 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 00:57:53.549292 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:53.549269 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-107.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 00:57:53.549761 ip-10-0-135-107 systemd[1]: Started Kubernetes Kubelet. Sep 11 00:57:53.552232 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.549797 2554 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 00:57:53.553313 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.552353 2554 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 00:57:53.553948 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.553927 2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jqkb9" Sep 11 00:57:53.555598 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:53.555520 2554 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 00:57:53.556274 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.556247 2554 server.go:310] "Adding debug handlers to kubelet server" Sep 11 00:57:53.557506 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:53.556586 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-107.ec2.internal.18d41e4e48b31510 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-107.ec2.internal,UID:ip-10-0-135-107.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-107.ec2.internal,},FirstTimestamp:2026-09-11 00:57:53.548907792 +0000 UTC m=+0.372361294,LastTimestamp:2026-09-11 00:57:53.548907792 +0000 UTC m=+0.372361294,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-107.ec2.internal,}" Sep 11 00:57:53.558349 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.558329 2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 00:57:53.559007 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.558988 2554 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 00:57:53.559119 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:53.559105 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-107.ec2.internal\" not found" Sep 11 00:57:53.559587 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.559571 2554 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 00:57:53.559587 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.559573 2554 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 00:57:53.559704 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.559598 2554 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 00:57:53.559766 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.559715 2554 reconstruct.go:97] "Volume reconstruction finished" Sep 11 00:57:53.559766 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.559725 2554 reconciler.go:29] "Reconciler: start to sync state" Sep 11 00:57:53.559887 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.559783 2554 factory.go:55] Registering systemd factory Sep 11 00:57:53.559887 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.559796 2554 factory.go:223] Registration of the systemd container factory successfully Sep 11 00:57:53.560005 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.559992 2554 factory.go:153] Registering CRI-O factory Sep 11 00:57:53.560005 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.560005 2554 factory.go:223] Registration of the crio container factory successfully Sep 11 00:57:53.560115 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.560046 2554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 00:57:53.560115 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.560059 2554 factory.go:103] Registering Raw factory Sep 11 00:57:53.560115 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.560070 2554 manager.go:1196] Started watching for new ooms in manager Sep 11 00:57:53.560522 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.560511 2554 manager.go:349] Starting recovery of all containers Sep 11 00:57:53.562416 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.562389 2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jqkb9" Sep 11 00:57:53.564650 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:53.564621 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 11 00:57:53.564773 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:53.564752 2554 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-107.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 11 00:57:53.570810 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.570788 2554 manager.go:354] Recovery completed Sep 11 00:57:53.576876 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.576864 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 00:57:53.578640 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.578623 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-107.ec2.internal" event="NodeHasSufficientMemory" Sep 11 00:57:53.578718 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.578652 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-107.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 00:57:53.578718 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.578663 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-107.ec2.internal" event="NodeHasSufficientPID" Sep 11 00:57:53.579237 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.579219 2554 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 00:57:53.579237 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.579236 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 00:57:53.579338 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.579237 2554 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 00:57:53.579338 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.579296 2554 state_mem.go:36] "Initialized new in-memory state store" Sep 11 00:57:53.579979 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.579963 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-107.ec2.internal" event="NodeHasSufficientMemory" Sep 11 00:57:53.580062 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.579990 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-107.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 00:57:53.580062 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.580003 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-107.ec2.internal" event="NodeHasSufficientPID" Sep 11 00:57:53.581207 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.581195 2554 policy_none.go:49] "None policy: Start" Sep 11 00:57:53.581255 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.581210 2554 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 00:57:53.581255 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.581220 2554 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 00:57:53.583068 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.583057 2554 policy_none.go:47] "Start" Sep 11 00:57:53.733204 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.620457 2554 manager.go:337] "Starting Device Plugin manager" Sep 11 00:57:53.733204 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:53.620486 2554 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 00:57:53.733204 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.620495 2554 server.go:87] "Starting device plugin registration server" Sep 11 00:57:53.733204 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.620664 2554 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 00:57:53.733204 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.620676 2554 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 00:57:53.733204 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.620797 2554 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 00:57:53.733204 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.620881 2554 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 00:57:53.733204 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.620892 2554 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 00:57:53.733204 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:53.621369 2554 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 00:57:53.733204 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:53.621419 2554 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-107.ec2.internal\" not found" Sep 11 00:57:53.733204 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.670093 2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 00:57:53.733204 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.671244 2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 00:57:53.733204 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.671289 2554 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 00:57:53.733204 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.671308 2554 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 00:57:53.733204 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:53.671347 2554 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 00:57:53.733204 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.673697 2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 00:57:53.733204 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.721738 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 00:57:53.733204 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.723173 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-107.ec2.internal" event="NodeHasSufficientMemory" Sep 11 00:57:53.733204 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.723202 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-107.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 00:57:53.733204 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.723214 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-107.ec2.internal" event="NodeHasSufficientPID" Sep 11 00:57:53.733204 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.723235 2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-107.ec2.internal" Sep 11 00:57:53.733204 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.730314 2554 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-107.ec2.internal" Sep 11 00:57:53.733204 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:53.730330 2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-107.ec2.internal\": node \"ip-10-0-135-107.ec2.internal\" not found" Sep 11 00:57:53.746960 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:53.746939 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-107.ec2.internal\" not found" Sep 11 00:57:53.771885 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.771864 2554 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-107.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-107.ec2.internal"] Sep 11 00:57:53.771953 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.771917 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 00:57:53.774399 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.774385 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-107.ec2.internal" event="NodeHasSufficientMemory" Sep 11 00:57:53.774465 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.774407 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-107.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 00:57:53.774465 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.774421 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-107.ec2.internal" event="NodeHasSufficientPID" Sep 11 00:57:53.776723 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.776711 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 00:57:53.776866 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.776852 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-107.ec2.internal" Sep 11 00:57:53.776916 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.776878 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 00:57:53.777588 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.777574 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-107.ec2.internal" event="NodeHasSufficientMemory" Sep 11 00:57:53.777666 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.777591 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-107.ec2.internal" event="NodeHasSufficientMemory" Sep 11 00:57:53.777666 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.777602 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-107.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 00:57:53.777666 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.777610 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-107.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 00:57:53.777666 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.777617 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-107.ec2.internal" event="NodeHasSufficientPID" Sep 11 00:57:53.777859 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.777619 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-107.ec2.internal" event="NodeHasSufficientPID" Sep 11 00:57:53.779800 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.779782 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-107.ec2.internal" Sep 11 00:57:53.779890 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.779810 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 00:57:53.780977 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.780964 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-107.ec2.internal" event="NodeHasSufficientMemory" Sep 11 00:57:53.781057 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.780990 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-107.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 00:57:53.781057 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.781003 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-107.ec2.internal" event="NodeHasSufficientPID" Sep 11 00:57:53.812387 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:53.812368 2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-107.ec2.internal\" not found" node="ip-10-0-135-107.ec2.internal" Sep 11 00:57:53.816644 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:53.816629 2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-107.ec2.internal\" not found" node="ip-10-0-135-107.ec2.internal" Sep 11 00:57:53.847819 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:53.847788 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-107.ec2.internal\" not found" Sep 11 00:57:53.861026 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.861007 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a1b1c28ec753bb4b358049fd0c5cad38-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-107.ec2.internal\" (UID: \"a1b1c28ec753bb4b358049fd0c5cad38\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-107.ec2.internal" Sep 11 00:57:53.861112 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.861027 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1b1c28ec753bb4b358049fd0c5cad38-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-107.ec2.internal\" (UID: \"a1b1c28ec753bb4b358049fd0c5cad38\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-107.ec2.internal" Sep 11 00:57:53.861112 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.861051 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/81a39435d308b5114d55354c422903c5-config\") pod \"kube-apiserver-proxy-ip-10-0-135-107.ec2.internal\" (UID: \"81a39435d308b5114d55354c422903c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-107.ec2.internal" Sep 11 00:57:53.948093 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:53.948065 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-107.ec2.internal\" not found" Sep 11 00:57:53.961436 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.961414 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a1b1c28ec753bb4b358049fd0c5cad38-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-107.ec2.internal\" (UID: \"a1b1c28ec753bb4b358049fd0c5cad38\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-107.ec2.internal" Sep 11 00:57:53.961498 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.961446 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1b1c28ec753bb4b358049fd0c5cad38-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-107.ec2.internal\" (UID: \"a1b1c28ec753bb4b358049fd0c5cad38\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-107.ec2.internal" Sep 11 00:57:53.961498 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.961475 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/81a39435d308b5114d55354c422903c5-config\") pod \"kube-apiserver-proxy-ip-10-0-135-107.ec2.internal\" (UID: \"81a39435d308b5114d55354c422903c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-107.ec2.internal" Sep 11 00:57:53.961498 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.961486 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a1b1c28ec753bb4b358049fd0c5cad38-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-107.ec2.internal\" (UID: \"a1b1c28ec753bb4b358049fd0c5cad38\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-107.ec2.internal" Sep 11 00:57:53.961620 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.961518 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/81a39435d308b5114d55354c422903c5-config\") pod \"kube-apiserver-proxy-ip-10-0-135-107.ec2.internal\" (UID: \"81a39435d308b5114d55354c422903c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-107.ec2.internal" Sep 11 00:57:53.961620 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:53.961527 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1b1c28ec753bb4b358049fd0c5cad38-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-107.ec2.internal\" (UID: \"a1b1c28ec753bb4b358049fd0c5cad38\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-107.ec2.internal" Sep 11 00:57:54.048817 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:54.048777 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-107.ec2.internal\" not found" Sep 11 00:57:54.115214 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.115192 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-107.ec2.internal" Sep 11 00:57:54.118781 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.118759 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-107.ec2.internal" Sep 11 00:57:54.149499 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:54.149355 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-107.ec2.internal\" not found" Sep 11 00:57:54.249632 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:54.249606 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-107.ec2.internal\" not found" Sep 11 00:57:54.350902 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:54.350831 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-107.ec2.internal\" not found" Sep 11 00:57:54.453008 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:54.452980 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-107.ec2.internal\" not found" Sep 11 00:57:54.464527 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.464506 2554 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 00:57:54.503931 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.503910 2554 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 00:57:54.504329 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.504019 2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 00:57:54.504329 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.504066 2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 00:57:54.539414 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.539392 2554 apiserver.go:52] "Watching apiserver" Sep 11 00:57:54.558346 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.558326 2554 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 00:57:54.558453 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.558439 2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 00:57:54.558805 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.558785 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-qxf8s","openshift-image-registry/node-ca-spz6s","openshift-network-diagnostics/network-check-target-dwb7w","openshift-network-operator/iptables-alerter-h9qsj","openshift-ovn-kubernetes/ovnkube-node-wtd8v","kube-system/konnectivity-agent-45g5g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf","openshift-multus/multus-additional-cni-plugins-kt8nl","openshift-multus/multus-bzv5j","openshift-multus/network-metrics-daemon-87kmh","kube-system/global-pull-secret-syncer-9rbhz","openshift-cluster-node-tuning-operator/tuned-8tgqd"] Sep 11 00:57:54.559366 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.559351 2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-107.ec2.internal" Sep 11 00:57:54.561617 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.561603 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qxf8s" Sep 11 00:57:54.563737 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.563699 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35c8be8a-e0b7-4843-b8db-4b6a486aee9e-hosts-file\") pod \"node-resolver-qxf8s\" (UID: \"35c8be8a-e0b7-4843-b8db-4b6a486aee9e\") " pod="openshift-dns/node-resolver-qxf8s" Sep 11 00:57:54.563828 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.563748 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c8be8a-e0b7-4843-b8db-4b6a486aee9e-tmp-dir\") pod \"node-resolver-qxf8s\" (UID: \"35c8be8a-e0b7-4843-b8db-4b6a486aee9e\") " pod="openshift-dns/node-resolver-qxf8s" Sep 11 00:57:54.563828 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.563787 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9zq9\" (UniqueName: \"kubernetes.io/projected/35c8be8a-e0b7-4843-b8db-4b6a486aee9e-kube-api-access-p9zq9\") pod \"node-resolver-qxf8s\" (UID: \"35c8be8a-e0b7-4843-b8db-4b6a486aee9e\") " pod="openshift-dns/node-resolver-qxf8s" Sep 11 00:57:54.563935 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.563849 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-spz6s" Sep 11 00:57:54.563994 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.563979 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l9lcz\"" Sep 11 00:57:54.564046 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.564029 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 00:57:54.564629 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.564515 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 00:57:54.566790 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.566729 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 00:57:54.566790 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.566717 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 00:52:53 +0000 UTC" deadline="2028-05-19 02:52:50.256771862 +0000 UTC" Sep 11 00:57:54.566790 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.566787 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14785h54m55.68999004s" Sep 11 00:57:54.566998 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.566769 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 00:57:54.566998 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.566852 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 00:57:54.567132 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.567112 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h5msj\"" Sep 11 00:57:54.567353 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.567337 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:57:54.567440 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:54.567418 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dwb7w" podUID="d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e" Sep 11 00:57:54.568875 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.568859 2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 00:57:54.568981 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.568925 2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-107.ec2.internal" Sep 11 00:57:54.569940 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.569922 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h9qsj" Sep 11 00:57:54.570997 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.570976 2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 00:57:54.572258 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.572217 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.572258 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.572257 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 00:57:54.572681 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.572661 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 00:57:54.572769 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.572724 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 00:57:54.572829 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.572803 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-z7kn5\"" Sep 11 00:57:54.574424 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.574407 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 00:57:54.574563 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.574544 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-45g5g" Sep 11 00:57:54.574669 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.574648 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cmg5c\"" Sep 11 00:57:54.574901 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.574848 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 00:57:54.575022 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.574981 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 00:57:54.575105 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.575072 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 00:57:54.575105 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.575074 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 00:57:54.575105 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.575098 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 00:57:54.576722 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.576701 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 00:57:54.576881 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.576704 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hccqm\"" Sep 11 00:57:54.577059 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.577042 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" Sep 11 00:57:54.577139 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.577072 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 00:57:54.579286 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.579265 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 00:57:54.579286 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.579280 2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 00:57:54.579451 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.579312 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 00:57:54.579451 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.579379 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d56n4\"" Sep 11 00:57:54.579659 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.579645 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 00:57:54.579983 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.579964 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kt8nl" Sep 11 00:57:54.581961 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.581943 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 00:57:54.582246 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.582231 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 00:57:54.582323 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.582277 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 00:57:54.582492 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.582476 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.582663 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.582647 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 00:57:54.582794 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.582779 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4kzqs\"" Sep 11 00:57:54.583240 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.583222 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 00:57:54.584602 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.584585 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xzpqj\"" Sep 11 00:57:54.584743 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.584726 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:57:54.584861 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:54.584823 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87kmh" podUID="8f0c9d36-ae86-4625-8dd1-0c9764c655e5" Sep 11 00:57:54.584925 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.584879 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 00:57:54.587242 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.587226 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:57:54.587301 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:54.587280 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rbhz" podUID="6038fbb3-56b0-49d0-875a-8f21cfdc9e77" Sep 11 00:57:54.590334 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.590316 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-107.ec2.internal"] Sep 11 00:57:54.590397 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.590351 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-107.ec2.internal"] Sep 11 00:57:54.590434 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.590409 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.593785 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.593768 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 00:57:54.593879 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.593767 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 00:57:54.593879 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.593771 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7ltgl\"" Sep 11 00:57:54.594225 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.594209 2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mvtfk" Sep 11 00:57:54.603827 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.603780 2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mvtfk" Sep 11 00:57:54.653921 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:54.653882 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b1c28ec753bb4b358049fd0c5cad38.slice/crio-c0812ae156d8bf4fb0a70918510c5fde9b03c0770b30e7078a804e2a7ff4f996 WatchSource:0}: Error finding container c0812ae156d8bf4fb0a70918510c5fde9b03c0770b30e7078a804e2a7ff4f996: Status 404 returned error can't find the container with id c0812ae156d8bf4fb0a70918510c5fde9b03c0770b30e7078a804e2a7ff4f996 Sep 11 00:57:54.654113 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:54.654100 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a39435d308b5114d55354c422903c5.slice/crio-d3e2cc3e312c9161a39a6c094b36e911be63b2d68a6623111c78afd23f229d62 WatchSource:0}: Error finding container d3e2cc3e312c9161a39a6c094b36e911be63b2d68a6623111c78afd23f229d62: Status 404 returned error can't find the container with id d3e2cc3e312c9161a39a6c094b36e911be63b2d68a6623111c78afd23f229d62 Sep 11 00:57:54.657852 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.657838 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 00:57:54.661328 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.661313 2554 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 00:57:54.664280 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664256 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c8be8a-e0b7-4843-b8db-4b6a486aee9e-tmp-dir\") pod \"node-resolver-qxf8s\" (UID: \"35c8be8a-e0b7-4843-b8db-4b6a486aee9e\") " pod="openshift-dns/node-resolver-qxf8s" Sep 11 00:57:54.664365 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664296 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d5568658-3c63-4779-86b5-df6dc73de2c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kt8nl\" (UID: \"d5568658-3c63-4779-86b5-df6dc73de2c7\") " pod="openshift-multus/multus-additional-cni-plugins-kt8nl" Sep 11 00:57:54.664365 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664326 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-etc-kubernetes\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.664448 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664429 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-host-run-netns\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.664485 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664454 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d5568658-3c63-4779-86b5-df6dc73de2c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kt8nl\" (UID: \"d5568658-3c63-4779-86b5-df6dc73de2c7\") " pod="openshift-multus/multus-additional-cni-plugins-kt8nl" Sep 11 00:57:54.664485 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664474 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqdl2\" (UniqueName: \"kubernetes.io/projected/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-kube-api-access-cqdl2\") pod \"network-metrics-daemon-87kmh\" (UID: \"8f0c9d36-ae86-4625-8dd1-0c9764c655e5\") " pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:57:54.664556 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664491 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-original-pull-secret\") pod \"global-pull-secret-syncer-9rbhz\" (UID: \"6038fbb3-56b0-49d0-875a-8f21cfdc9e77\") " pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:57:54.664556 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664516 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-etc-sysconfig\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.664556 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664528 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35c8be8a-e0b7-4843-b8db-4b6a486aee9e-tmp-dir\") pod \"node-resolver-qxf8s\" (UID: \"35c8be8a-e0b7-4843-b8db-4b6a486aee9e\") " pod="openshift-dns/node-resolver-qxf8s" Sep 11 00:57:54.664556 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664539 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ecf0bd8-8c32-4bcd-a586-039de22032ad-sys-fs\") pod \"aws-ebs-csi-driver-node-d55hf\" (UID: \"7ecf0bd8-8c32-4bcd-a586-039de22032ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" Sep 11 00:57:54.664556 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664553 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-run\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.664717 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664596 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tql7k\" (UniqueName: \"kubernetes.io/projected/b0d3c13b-c771-4648-bc19-cbea081e4e4c-kube-api-access-tql7k\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.664717 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664652 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm4jf\" (UniqueName: \"kubernetes.io/projected/70820178-37d7-4c9e-8240-0905cdfe37ee-kube-api-access-xm4jf\") pod \"node-ca-spz6s\" (UID: \"70820178-37d7-4c9e-8240-0905cdfe37ee\") " pod="openshift-image-registry/node-ca-spz6s" Sep 11 00:57:54.664717 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664688 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-host-slash\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.664814 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664715 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ecf0bd8-8c32-4bcd-a586-039de22032ad-socket-dir\") pod \"aws-ebs-csi-driver-node-d55hf\" (UID: \"7ecf0bd8-8c32-4bcd-a586-039de22032ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" Sep 11 00:57:54.664814 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664739 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-cni-binary-copy\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.664814 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664764 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-etc-kubernetes\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.664814 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664786 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-etc-modprobe-d\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.664969 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664814 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70820178-37d7-4c9e-8240-0905cdfe37ee-serviceca\") pod \"node-ca-spz6s\" (UID: \"70820178-37d7-4c9e-8240-0905cdfe37ee\") " pod="openshift-image-registry/node-ca-spz6s" Sep 11 00:57:54.664969 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664841 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr4v5\" (UniqueName: \"kubernetes.io/projected/7ecf0bd8-8c32-4bcd-a586-039de22032ad-kube-api-access-pr4v5\") pod \"aws-ebs-csi-driver-node-d55hf\" (UID: \"7ecf0bd8-8c32-4bcd-a586-039de22032ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" Sep 11 00:57:54.664969 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664880 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-host-run-k8s-cni-cncf-io\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.664969 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664895 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-host-var-lib-cni-bin\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.664969 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664909 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-host\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.664969 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664922 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-run-ovn\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.664969 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664941 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-host-run-ovn-kubernetes\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.664969 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664962 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-hostroot\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.665257 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.664989 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-host-run-multus-certs\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.665257 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665012 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpg9s\" (UniqueName: \"kubernetes.io/projected/edff204d-9959-4821-9aa8-a4c33f9b07b1-kube-api-access-vpg9s\") pod \"iptables-alerter-h9qsj\" (UID: \"edff204d-9959-4821-9aa8-a4c33f9b07b1\") " pod="openshift-network-operator/iptables-alerter-h9qsj" Sep 11 00:57:54.665257 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665032 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35c8be8a-e0b7-4843-b8db-4b6a486aee9e-hosts-file\") pod \"node-resolver-qxf8s\" (UID: \"35c8be8a-e0b7-4843-b8db-4b6a486aee9e\") " pod="openshift-dns/node-resolver-qxf8s" Sep 11 00:57:54.665257 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665068 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0d3c13b-c771-4648-bc19-cbea081e4e4c-tmp\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.665257 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665095 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78czw\" (UniqueName: \"kubernetes.io/projected/d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e-kube-api-access-78czw\") pod \"network-check-target-dwb7w\" (UID: \"d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e\") " pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:57:54.665257 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665110 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35c8be8a-e0b7-4843-b8db-4b6a486aee9e-hosts-file\") pod \"node-resolver-qxf8s\" (UID: \"35c8be8a-e0b7-4843-b8db-4b6a486aee9e\") " pod="openshift-dns/node-resolver-qxf8s" Sep 11 00:57:54.665257 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665121 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-log-socket\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.665257 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665145 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ecf0bd8-8c32-4bcd-a586-039de22032ad-device-dir\") pod \"aws-ebs-csi-driver-node-d55hf\" (UID: \"7ecf0bd8-8c32-4bcd-a586-039de22032ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" Sep 11 00:57:54.665257 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665164 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3794d977-c81b-4042-ad21-bd34d7462018-konnectivity-ca\") pod \"konnectivity-agent-45g5g\" (UID: \"3794d977-c81b-4042-ad21-bd34d7462018\") " pod="kube-system/konnectivity-agent-45g5g" Sep 11 00:57:54.665257 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665182 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5568658-3c63-4779-86b5-df6dc73de2c7-cnibin\") pod \"multus-additional-cni-plugins-kt8nl\" (UID: \"d5568658-3c63-4779-86b5-df6dc73de2c7\") " pod="openshift-multus/multus-additional-cni-plugins-kt8nl" Sep 11 00:57:54.665257 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665202 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-metrics-certs\") pod \"network-metrics-daemon-87kmh\" (UID: \"8f0c9d36-ae86-4625-8dd1-0c9764c655e5\") " pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:57:54.665257 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665226 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-kubelet-config\") pod \"global-pull-secret-syncer-9rbhz\" (UID: \"6038fbb3-56b0-49d0-875a-8f21cfdc9e77\") " pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:57:54.665257 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665242 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-var-lib-kubelet\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.665257 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665264 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-etc-openvswitch\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.665869 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665284 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-ovnkube-script-lib\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.665869 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665307 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-etc-sysctl-d\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.665869 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665333 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70820178-37d7-4c9e-8240-0905cdfe37ee-host\") pod \"node-ca-spz6s\" (UID: \"70820178-37d7-4c9e-8240-0905cdfe37ee\") " pod="openshift-image-registry/node-ca-spz6s" Sep 11 00:57:54.665869 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665351 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-host-kubelet\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.665869 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665371 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.665869 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665392 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ecf0bd8-8c32-4bcd-a586-039de22032ad-registration-dir\") pod \"aws-ebs-csi-driver-node-d55hf\" (UID: \"7ecf0bd8-8c32-4bcd-a586-039de22032ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" Sep 11 00:57:54.665869 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665412 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9zq9\" (UniqueName: \"kubernetes.io/projected/35c8be8a-e0b7-4843-b8db-4b6a486aee9e-kube-api-access-p9zq9\") pod \"node-resolver-qxf8s\" (UID: \"35c8be8a-e0b7-4843-b8db-4b6a486aee9e\") " pod="openshift-dns/node-resolver-qxf8s" Sep 11 00:57:54.665869 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665520 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5568658-3c63-4779-86b5-df6dc73de2c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-kt8nl\" (UID: \"d5568658-3c63-4779-86b5-df6dc73de2c7\") " pod="openshift-multus/multus-additional-cni-plugins-kt8nl" Sep 11 00:57:54.665869 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665547 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-dbus\") pod \"global-pull-secret-syncer-9rbhz\" (UID: \"6038fbb3-56b0-49d0-875a-8f21cfdc9e77\") " pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:57:54.665869 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665569 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-host-cni-bin\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.665869 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665591 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-ovnkube-config\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.665869 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665612 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-multus-cni-dir\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.665869 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665642 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnmc2\" (UniqueName: \"kubernetes.io/projected/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-kube-api-access-qnmc2\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.665869 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665661 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5568658-3c63-4779-86b5-df6dc73de2c7-system-cni-dir\") pod \"multus-additional-cni-plugins-kt8nl\" (UID: \"d5568658-3c63-4779-86b5-df6dc73de2c7\") " pod="openshift-multus/multus-additional-cni-plugins-kt8nl" Sep 11 00:57:54.665869 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665675 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-etc-systemd\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.665869 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665698 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-node-log\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.666552 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665721 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-host-cni-netd\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.666552 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665750 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-env-overrides\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.666552 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665769 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-run-systemd\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.666552 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665824 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-ovn-node-metrics-cert\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.666552 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665883 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ecf0bd8-8c32-4bcd-a586-039de22032ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d55hf\" (UID: \"7ecf0bd8-8c32-4bcd-a586-039de22032ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" Sep 11 00:57:54.666552 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665909 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-cnibin\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.666552 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665941 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-multus-daemon-config\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.666552 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665964 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b0d3c13b-c771-4648-bc19-cbea081e4e4c-etc-tuned\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.666552 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.665991 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-var-lib-openvswitch\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.666552 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.666014 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3794d977-c81b-4042-ad21-bd34d7462018-agent-certs\") pod \"konnectivity-agent-45g5g\" (UID: \"3794d977-c81b-4042-ad21-bd34d7462018\") " pod="kube-system/konnectivity-agent-45g5g" Sep 11 00:57:54.666552 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.666038 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2sq6\" (UniqueName: \"kubernetes.io/projected/d5568658-3c63-4779-86b5-df6dc73de2c7-kube-api-access-n2sq6\") pod \"multus-additional-cni-plugins-kt8nl\" (UID: \"d5568658-3c63-4779-86b5-df6dc73de2c7\") " pod="openshift-multus/multus-additional-cni-plugins-kt8nl" Sep 11 00:57:54.666552 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.666063 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-lib-modules\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.666552 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.666086 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-systemd-units\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.666552 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.666110 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ecf0bd8-8c32-4bcd-a586-039de22032ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-d55hf\" (UID: \"7ecf0bd8-8c32-4bcd-a586-039de22032ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" Sep 11 00:57:54.666552 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.666148 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-multus-socket-dir-parent\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.666552 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.666183 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-host-run-netns\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.667081 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.666212 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-run-openvswitch\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.667081 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.666236 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-host-var-lib-cni-multus\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.667081 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.666259 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-host-var-lib-kubelet\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.667081 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.666284 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/edff204d-9959-4821-9aa8-a4c33f9b07b1-iptables-alerter-script\") pod \"iptables-alerter-h9qsj\" (UID: \"edff204d-9959-4821-9aa8-a4c33f9b07b1\") " pod="openshift-network-operator/iptables-alerter-h9qsj" Sep 11 00:57:54.667081 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.666309 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/edff204d-9959-4821-9aa8-a4c33f9b07b1-host-slash\") pod \"iptables-alerter-h9qsj\" (UID: \"edff204d-9959-4821-9aa8-a4c33f9b07b1\") " pod="openshift-network-operator/iptables-alerter-h9qsj" Sep 11 00:57:54.667081 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.666332 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5568658-3c63-4779-86b5-df6dc73de2c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kt8nl\" (UID: \"d5568658-3c63-4779-86b5-df6dc73de2c7\") " pod="openshift-multus/multus-additional-cni-plugins-kt8nl" Sep 11 00:57:54.667081 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.666359 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-etc-sysctl-conf\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.667081 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.666381 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-sys\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.667081 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.666403 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-multus-conf-dir\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.667081 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.666427 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzc25\" (UniqueName: \"kubernetes.io/projected/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-kube-api-access-wzc25\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.667081 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.666453 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-system-cni-dir\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.671342 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.671307 2554 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 00:57:54.674626 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.674579 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-107.ec2.internal" event={"ID":"81a39435d308b5114d55354c422903c5","Type":"ContainerStarted","Data":"d3e2cc3e312c9161a39a6c094b36e911be63b2d68a6623111c78afd23f229d62"} Sep 11 00:57:54.675130 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.675108 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9zq9\" (UniqueName: \"kubernetes.io/projected/35c8be8a-e0b7-4843-b8db-4b6a486aee9e-kube-api-access-p9zq9\") pod \"node-resolver-qxf8s\" (UID: \"35c8be8a-e0b7-4843-b8db-4b6a486aee9e\") " pod="openshift-dns/node-resolver-qxf8s" Sep 11 00:57:54.676290 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.676271 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-107.ec2.internal" event={"ID":"a1b1c28ec753bb4b358049fd0c5cad38","Type":"ContainerStarted","Data":"c0812ae156d8bf4fb0a70918510c5fde9b03c0770b30e7078a804e2a7ff4f996"} Sep 11 00:57:54.766681 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.766653 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-node-log\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.766681 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.766681 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-host-cni-netd\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.766840 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.766697 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-env-overrides\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.766840 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.766711 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-run-systemd\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.766840 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.766726 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-ovn-node-metrics-cert\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.766840 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.766748 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ecf0bd8-8c32-4bcd-a586-039de22032ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d55hf\" (UID: \"7ecf0bd8-8c32-4bcd-a586-039de22032ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" Sep 11 00:57:54.766840 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.766771 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-cnibin\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.766840 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.766772 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-host-cni-netd\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.766840 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.766780 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-node-log\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.766840 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.766793 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-multus-daemon-config\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.766840 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.766792 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-run-systemd\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.766840 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.766829 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ecf0bd8-8c32-4bcd-a586-039de22032ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d55hf\" (UID: \"7ecf0bd8-8c32-4bcd-a586-039de22032ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" Sep 11 00:57:54.767364 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.766862 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-cnibin\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.767364 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.766890 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b0d3c13b-c771-4648-bc19-cbea081e4e4c-etc-tuned\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.767364 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.766912 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-var-lib-openvswitch\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.767364 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.766928 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3794d977-c81b-4042-ad21-bd34d7462018-agent-certs\") pod \"konnectivity-agent-45g5g\" (UID: \"3794d977-c81b-4042-ad21-bd34d7462018\") " pod="kube-system/konnectivity-agent-45g5g" Sep 11 00:57:54.767364 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.766960 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2sq6\" (UniqueName: \"kubernetes.io/projected/d5568658-3c63-4779-86b5-df6dc73de2c7-kube-api-access-n2sq6\") pod \"multus-additional-cni-plugins-kt8nl\" (UID: \"d5568658-3c63-4779-86b5-df6dc73de2c7\") " pod="openshift-multus/multus-additional-cni-plugins-kt8nl" Sep 11 00:57:54.767364 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.766984 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-var-lib-openvswitch\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.767364 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767075 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-lib-modules\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.767364 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.766987 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-lib-modules\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.767364 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767114 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-systemd-units\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.767364 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767140 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ecf0bd8-8c32-4bcd-a586-039de22032ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-d55hf\" (UID: \"7ecf0bd8-8c32-4bcd-a586-039de22032ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" Sep 11 00:57:54.767364 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767165 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-multus-socket-dir-parent\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.767364 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767188 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-host-run-netns\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.767364 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767211 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-run-openvswitch\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.767364 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767236 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-host-var-lib-cni-multus\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.767364 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767258 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ecf0bd8-8c32-4bcd-a586-039de22032ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-d55hf\" (UID: \"7ecf0bd8-8c32-4bcd-a586-039de22032ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" Sep 11 00:57:54.767364 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767260 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-host-var-lib-kubelet\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.767364 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767299 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/edff204d-9959-4821-9aa8-a4c33f9b07b1-iptables-alerter-script\") pod \"iptables-alerter-h9qsj\" (UID: \"edff204d-9959-4821-9aa8-a4c33f9b07b1\") " pod="openshift-network-operator/iptables-alerter-h9qsj" Sep 11 00:57:54.768237 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767304 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-host-var-lib-kubelet\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.768237 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767333 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/edff204d-9959-4821-9aa8-a4c33f9b07b1-host-slash\") pod \"iptables-alerter-h9qsj\" (UID: \"edff204d-9959-4821-9aa8-a4c33f9b07b1\") " pod="openshift-network-operator/iptables-alerter-h9qsj" Sep 11 00:57:54.768237 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767348 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-systemd-units\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.768237 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767358 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5568658-3c63-4779-86b5-df6dc73de2c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kt8nl\" (UID: \"d5568658-3c63-4779-86b5-df6dc73de2c7\") " pod="openshift-multus/multus-additional-cni-plugins-kt8nl" Sep 11 00:57:54.768237 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767385 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-etc-sysctl-conf\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.768237 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767417 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-multus-daemon-config\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.768237 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767431 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/edff204d-9959-4821-9aa8-a4c33f9b07b1-host-slash\") pod \"iptables-alerter-h9qsj\" (UID: \"edff204d-9959-4821-9aa8-a4c33f9b07b1\") " pod="openshift-network-operator/iptables-alerter-h9qsj" Sep 11 00:57:54.768237 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767446 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-multus-socket-dir-parent\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.768237 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767484 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-sys\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.768237 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767488 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-etc-sysctl-conf\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.768237 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767525 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-run-openvswitch\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.768237 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767629 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-multus-conf-dir\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.768237 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767656 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzc25\" (UniqueName: \"kubernetes.io/projected/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-kube-api-access-wzc25\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.768237 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767682 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-system-cni-dir\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.768237 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767727 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5568658-3c63-4779-86b5-df6dc73de2c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kt8nl\" (UID: \"d5568658-3c63-4779-86b5-df6dc73de2c7\") " pod="openshift-multus/multus-additional-cni-plugins-kt8nl" Sep 11 00:57:54.768237 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767739 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d5568658-3c63-4779-86b5-df6dc73de2c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kt8nl\" (UID: \"d5568658-3c63-4779-86b5-df6dc73de2c7\") " pod="openshift-multus/multus-additional-cni-plugins-kt8nl" Sep 11 00:57:54.768237 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767767 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-etc-kubernetes\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.769060 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767765 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-env-overrides\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.769060 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767772 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-host-var-lib-cni-multus\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.769060 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767802 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-multus-conf-dir\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.769060 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767831 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-host-run-netns\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.769060 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767863 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d5568658-3c63-4779-86b5-df6dc73de2c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kt8nl\" (UID: \"d5568658-3c63-4779-86b5-df6dc73de2c7\") " pod="openshift-multus/multus-additional-cni-plugins-kt8nl" Sep 11 00:57:54.769060 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767833 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-etc-kubernetes\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.769060 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767891 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-sys\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.769060 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767964 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-host-run-netns\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.769060 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.768003 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqdl2\" (UniqueName: \"kubernetes.io/projected/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-kube-api-access-cqdl2\") pod \"network-metrics-daemon-87kmh\" (UID: \"8f0c9d36-ae86-4625-8dd1-0c9764c655e5\") " pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:57:54.769060 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.768023 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/edff204d-9959-4821-9aa8-a4c33f9b07b1-iptables-alerter-script\") pod \"iptables-alerter-h9qsj\" (UID: \"edff204d-9959-4821-9aa8-a4c33f9b07b1\") " pod="openshift-network-operator/iptables-alerter-h9qsj" Sep 11 00:57:54.769060 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.768058 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-original-pull-secret\") pod \"global-pull-secret-syncer-9rbhz\" (UID: \"6038fbb3-56b0-49d0-875a-8f21cfdc9e77\") " pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:57:54.769060 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.768066 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-system-cni-dir\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.769060 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.768141 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-etc-sysconfig\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.769060 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:54.768157 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 00:57:54.769060 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.768168 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ecf0bd8-8c32-4bcd-a586-039de22032ad-sys-fs\") pod \"aws-ebs-csi-driver-node-d55hf\" (UID: \"7ecf0bd8-8c32-4bcd-a586-039de22032ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" Sep 11 00:57:54.769060 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.768192 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-run\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.769060 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.768217 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tql7k\" (UniqueName: \"kubernetes.io/projected/b0d3c13b-c771-4648-bc19-cbea081e4e4c-kube-api-access-tql7k\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.770024 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:54.768249 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-original-pull-secret podName:6038fbb3-56b0-49d0-875a-8f21cfdc9e77 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:55.268226914 +0000 UTC m=+2.091680420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-original-pull-secret") pod "global-pull-secret-syncer-9rbhz" (UID: "6038fbb3-56b0-49d0-875a-8f21cfdc9e77") : object "kube-system"/"original-pull-secret" not registered Sep 11 00:57:54.770024 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.768278 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ecf0bd8-8c32-4bcd-a586-039de22032ad-sys-fs\") pod \"aws-ebs-csi-driver-node-d55hf\" (UID: \"7ecf0bd8-8c32-4bcd-a586-039de22032ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" Sep 11 00:57:54.770024 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.768269 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm4jf\" (UniqueName: \"kubernetes.io/projected/70820178-37d7-4c9e-8240-0905cdfe37ee-kube-api-access-xm4jf\") pod \"node-ca-spz6s\" (UID: \"70820178-37d7-4c9e-8240-0905cdfe37ee\") " pod="openshift-image-registry/node-ca-spz6s" Sep 11 00:57:54.770024 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.768322 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-run\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.770024 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.768424 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d5568658-3c63-4779-86b5-df6dc73de2c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kt8nl\" (UID: \"d5568658-3c63-4779-86b5-df6dc73de2c7\") " pod="openshift-multus/multus-additional-cni-plugins-kt8nl" Sep 11 00:57:54.770024 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.768476 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-host-slash\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.770024 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.768531 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d5568658-3c63-4779-86b5-df6dc73de2c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kt8nl\" (UID: \"d5568658-3c63-4779-86b5-df6dc73de2c7\") " pod="openshift-multus/multus-additional-cni-plugins-kt8nl" Sep 11 00:57:54.770024 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.767583 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-host-run-netns\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.770024 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.768611 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ecf0bd8-8c32-4bcd-a586-039de22032ad-socket-dir\") pod \"aws-ebs-csi-driver-node-d55hf\" (UID: \"7ecf0bd8-8c32-4bcd-a586-039de22032ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" Sep 11 00:57:54.770024 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.768607 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-etc-sysconfig\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.770024 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.768662 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-cni-binary-copy\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.770024 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.768695 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-etc-kubernetes\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.770024 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.768727 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-host-slash\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.770024 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.768729 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ecf0bd8-8c32-4bcd-a586-039de22032ad-socket-dir\") pod \"aws-ebs-csi-driver-node-d55hf\" (UID: \"7ecf0bd8-8c32-4bcd-a586-039de22032ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" Sep 11 00:57:54.770024 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.768737 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-etc-modprobe-d\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.770024 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769030 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70820178-37d7-4c9e-8240-0905cdfe37ee-serviceca\") pod \"node-ca-spz6s\" (UID: \"70820178-37d7-4c9e-8240-0905cdfe37ee\") " pod="openshift-image-registry/node-ca-spz6s" Sep 11 00:57:54.770767 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769060 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr4v5\" (UniqueName: \"kubernetes.io/projected/7ecf0bd8-8c32-4bcd-a586-039de22032ad-kube-api-access-pr4v5\") pod \"aws-ebs-csi-driver-node-d55hf\" (UID: \"7ecf0bd8-8c32-4bcd-a586-039de22032ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" Sep 11 00:57:54.770767 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769088 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-host-run-k8s-cni-cncf-io\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.770767 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769114 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-host-var-lib-cni-bin\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.770767 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769118 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-cni-binary-copy\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.770767 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769136 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-host\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.770767 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769160 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-run-ovn\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.770767 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769183 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-host-run-ovn-kubernetes\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.770767 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769205 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-hostroot\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.770767 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769229 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-host-run-multus-certs\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.770767 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.768864 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-etc-modprobe-d\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.770767 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.768765 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-etc-kubernetes\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.770767 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769308 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpg9s\" (UniqueName: \"kubernetes.io/projected/edff204d-9959-4821-9aa8-a4c33f9b07b1-kube-api-access-vpg9s\") pod \"iptables-alerter-h9qsj\" (UID: \"edff204d-9959-4821-9aa8-a4c33f9b07b1\") " pod="openshift-network-operator/iptables-alerter-h9qsj" Sep 11 00:57:54.770767 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769321 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-host-run-multus-certs\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.770767 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769348 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-host-run-ovn-kubernetes\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.770767 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769345 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0d3c13b-c771-4648-bc19-cbea081e4e4c-tmp\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.770767 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769406 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-run-ovn\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.770767 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769367 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-host\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.770767 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769408 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78czw\" (UniqueName: \"kubernetes.io/projected/d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e-kube-api-access-78czw\") pod \"network-check-target-dwb7w\" (UID: \"d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e\") " pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:57:54.771455 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769412 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-host-var-lib-cni-bin\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.771455 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769446 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-host-run-k8s-cni-cncf-io\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.771455 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769451 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-log-socket\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.771455 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769366 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-hostroot\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.771455 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769479 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ecf0bd8-8c32-4bcd-a586-039de22032ad-device-dir\") pod \"aws-ebs-csi-driver-node-d55hf\" (UID: \"7ecf0bd8-8c32-4bcd-a586-039de22032ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" Sep 11 00:57:54.771455 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769498 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70820178-37d7-4c9e-8240-0905cdfe37ee-serviceca\") pod \"node-ca-spz6s\" (UID: \"70820178-37d7-4c9e-8240-0905cdfe37ee\") " pod="openshift-image-registry/node-ca-spz6s" Sep 11 00:57:54.771455 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769532 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ecf0bd8-8c32-4bcd-a586-039de22032ad-device-dir\") pod \"aws-ebs-csi-driver-node-d55hf\" (UID: \"7ecf0bd8-8c32-4bcd-a586-039de22032ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" Sep 11 00:57:54.771455 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769544 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-log-socket\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.771455 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769597 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3794d977-c81b-4042-ad21-bd34d7462018-konnectivity-ca\") pod \"konnectivity-agent-45g5g\" (UID: \"3794d977-c81b-4042-ad21-bd34d7462018\") " pod="kube-system/konnectivity-agent-45g5g" Sep 11 00:57:54.771455 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769661 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5568658-3c63-4779-86b5-df6dc73de2c7-cnibin\") pod \"multus-additional-cni-plugins-kt8nl\" (UID: \"d5568658-3c63-4779-86b5-df6dc73de2c7\") " pod="openshift-multus/multus-additional-cni-plugins-kt8nl" Sep 11 00:57:54.771455 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769706 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-metrics-certs\") pod \"network-metrics-daemon-87kmh\" (UID: \"8f0c9d36-ae86-4625-8dd1-0c9764c655e5\") " pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:57:54.771455 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769732 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-kubelet-config\") pod \"global-pull-secret-syncer-9rbhz\" (UID: \"6038fbb3-56b0-49d0-875a-8f21cfdc9e77\") " pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:57:54.771455 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769794 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b0d3c13b-c771-4648-bc19-cbea081e4e4c-etc-tuned\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.771455 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:54.769810 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 00:57:54.771455 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769807 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-ovn-node-metrics-cert\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.771455 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:54.769859 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-metrics-certs podName:8f0c9d36-ae86-4625-8dd1-0c9764c655e5 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:55.269842398 +0000 UTC m=+2.093295902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-metrics-certs") pod "network-metrics-daemon-87kmh" (UID: "8f0c9d36-ae86-4625-8dd1-0c9764c655e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 00:57:54.771455 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769887 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-kubelet-config\") pod \"global-pull-secret-syncer-9rbhz\" (UID: \"6038fbb3-56b0-49d0-875a-8f21cfdc9e77\") " pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:57:54.772044 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769916 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5568658-3c63-4779-86b5-df6dc73de2c7-cnibin\") pod \"multus-additional-cni-plugins-kt8nl\" (UID: \"d5568658-3c63-4779-86b5-df6dc73de2c7\") " pod="openshift-multus/multus-additional-cni-plugins-kt8nl" Sep 11 00:57:54.772044 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769942 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-var-lib-kubelet\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.772044 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769971 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-etc-openvswitch\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.772044 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769987 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3794d977-c81b-4042-ad21-bd34d7462018-agent-certs\") pod \"konnectivity-agent-45g5g\" (UID: \"3794d977-c81b-4042-ad21-bd34d7462018\") " pod="kube-system/konnectivity-agent-45g5g" Sep 11 00:57:54.772044 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.769996 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-ovnkube-script-lib\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.772044 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770006 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-var-lib-kubelet\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.772044 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770031 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-etc-sysctl-d\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.772044 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770045 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-etc-openvswitch\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.772044 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770055 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70820178-37d7-4c9e-8240-0905cdfe37ee-host\") pod \"node-ca-spz6s\" (UID: \"70820178-37d7-4c9e-8240-0905cdfe37ee\") " pod="openshift-image-registry/node-ca-spz6s" Sep 11 00:57:54.772044 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770103 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70820178-37d7-4c9e-8240-0905cdfe37ee-host\") pod \"node-ca-spz6s\" (UID: \"70820178-37d7-4c9e-8240-0905cdfe37ee\") " pod="openshift-image-registry/node-ca-spz6s" Sep 11 00:57:54.772044 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770127 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3794d977-c81b-4042-ad21-bd34d7462018-konnectivity-ca\") pod \"konnectivity-agent-45g5g\" (UID: \"3794d977-c81b-4042-ad21-bd34d7462018\") " pod="kube-system/konnectivity-agent-45g5g" Sep 11 00:57:54.772044 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770133 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-host-kubelet\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.772044 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770155 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-etc-sysctl-d\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.772044 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770165 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-host-kubelet\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.772044 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770186 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.772044 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770218 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ecf0bd8-8c32-4bcd-a586-039de22032ad-registration-dir\") pod \"aws-ebs-csi-driver-node-d55hf\" (UID: \"7ecf0bd8-8c32-4bcd-a586-039de22032ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" Sep 11 00:57:54.772044 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770224 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.772633 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770246 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5568658-3c63-4779-86b5-df6dc73de2c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-kt8nl\" (UID: \"d5568658-3c63-4779-86b5-df6dc73de2c7\") " pod="openshift-multus/multus-additional-cni-plugins-kt8nl" Sep 11 00:57:54.772633 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770271 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-dbus\") pod \"global-pull-secret-syncer-9rbhz\" (UID: \"6038fbb3-56b0-49d0-875a-8f21cfdc9e77\") " pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:57:54.772633 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770279 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ecf0bd8-8c32-4bcd-a586-039de22032ad-registration-dir\") pod \"aws-ebs-csi-driver-node-d55hf\" (UID: \"7ecf0bd8-8c32-4bcd-a586-039de22032ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" Sep 11 00:57:54.772633 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770296 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-host-cni-bin\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.772633 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770320 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-ovnkube-config\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.772633 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770346 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-multus-cni-dir\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.772633 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770375 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnmc2\" (UniqueName: \"kubernetes.io/projected/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-kube-api-access-qnmc2\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.772633 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770402 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5568658-3c63-4779-86b5-df6dc73de2c7-system-cni-dir\") pod \"multus-additional-cni-plugins-kt8nl\" (UID: \"d5568658-3c63-4779-86b5-df6dc73de2c7\") " pod="openshift-multus/multus-additional-cni-plugins-kt8nl" Sep 11 00:57:54.772633 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770428 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-etc-systemd\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.772633 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770427 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-dbus\") pod \"global-pull-secret-syncer-9rbhz\" (UID: \"6038fbb3-56b0-49d0-875a-8f21cfdc9e77\") " pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:57:54.772633 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770474 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-ovnkube-script-lib\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.772633 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770527 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5568658-3c63-4779-86b5-df6dc73de2c7-system-cni-dir\") pod \"multus-additional-cni-plugins-kt8nl\" (UID: \"d5568658-3c63-4779-86b5-df6dc73de2c7\") " pod="openshift-multus/multus-additional-cni-plugins-kt8nl" Sep 11 00:57:54.772633 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770572 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b0d3c13b-c771-4648-bc19-cbea081e4e4c-etc-systemd\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.772633 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770626 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-multus-cni-dir\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.772633 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770690 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5568658-3c63-4779-86b5-df6dc73de2c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-kt8nl\" (UID: \"d5568658-3c63-4779-86b5-df6dc73de2c7\") " pod="openshift-multus/multus-additional-cni-plugins-kt8nl" Sep 11 00:57:54.772633 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770824 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-ovnkube-config\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.772633 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.770927 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-host-cni-bin\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.773297 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.771245 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0d3c13b-c771-4648-bc19-cbea081e4e4c-tmp\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.775799 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:54.775783 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 00:57:54.775799 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:54.775801 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 00:57:54.775984 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:54.775810 2554 projected.go:196] Error preparing data for projected volume kube-api-access-78czw for pod openshift-network-diagnostics/network-check-target-dwb7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 00:57:54.775984 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:54.775857 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e-kube-api-access-78czw podName:d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e nodeName:}" failed. No retries permitted until 2026-09-11 00:57:55.275846697 +0000 UTC m=+2.099300189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-78czw" (UniqueName: "kubernetes.io/projected/d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e-kube-api-access-78czw") pod "network-check-target-dwb7w" (UID: "d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 00:57:54.776765 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.776741 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2sq6\" (UniqueName: \"kubernetes.io/projected/d5568658-3c63-4779-86b5-df6dc73de2c7-kube-api-access-n2sq6\") pod \"multus-additional-cni-plugins-kt8nl\" (UID: \"d5568658-3c63-4779-86b5-df6dc73de2c7\") " pod="openshift-multus/multus-additional-cni-plugins-kt8nl" Sep 11 00:57:54.778694 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.778667 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzc25\" (UniqueName: \"kubernetes.io/projected/ac49dda0-6ce3-42d9-83e7-1e22ffa2a096-kube-api-access-wzc25\") pod \"multus-bzv5j\" (UID: \"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096\") " pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.778800 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.778783 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpg9s\" (UniqueName: \"kubernetes.io/projected/edff204d-9959-4821-9aa8-a4c33f9b07b1-kube-api-access-vpg9s\") pod \"iptables-alerter-h9qsj\" (UID: \"edff204d-9959-4821-9aa8-a4c33f9b07b1\") " pod="openshift-network-operator/iptables-alerter-h9qsj" Sep 11 00:57:54.778902 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.778877 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm4jf\" (UniqueName: \"kubernetes.io/projected/70820178-37d7-4c9e-8240-0905cdfe37ee-kube-api-access-xm4jf\") pod \"node-ca-spz6s\" (UID: \"70820178-37d7-4c9e-8240-0905cdfe37ee\") " pod="openshift-image-registry/node-ca-spz6s" Sep 11 00:57:54.779081 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.779060 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tql7k\" (UniqueName: \"kubernetes.io/projected/b0d3c13b-c771-4648-bc19-cbea081e4e4c-kube-api-access-tql7k\") pod \"tuned-8tgqd\" (UID: \"b0d3c13b-c771-4648-bc19-cbea081e4e4c\") " pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.779195 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.779160 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqdl2\" (UniqueName: \"kubernetes.io/projected/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-kube-api-access-cqdl2\") pod \"network-metrics-daemon-87kmh\" (UID: \"8f0c9d36-ae86-4625-8dd1-0c9764c655e5\") " pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:57:54.779333 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.779311 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr4v5\" (UniqueName: \"kubernetes.io/projected/7ecf0bd8-8c32-4bcd-a586-039de22032ad-kube-api-access-pr4v5\") pod \"aws-ebs-csi-driver-node-d55hf\" (UID: \"7ecf0bd8-8c32-4bcd-a586-039de22032ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" Sep 11 00:57:54.779447 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.779430 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnmc2\" (UniqueName: \"kubernetes.io/projected/c2b320dd-43e9-435d-8e4f-2bca1f1233ff-kube-api-access-qnmc2\") pod \"ovnkube-node-wtd8v\" (UID: \"c2b320dd-43e9-435d-8e4f-2bca1f1233ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.878970 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.878921 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qxf8s" Sep 11 00:57:54.889348 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:54.889327 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c8be8a_e0b7_4843_b8db_4b6a486aee9e.slice/crio-ae515a6cf138a51736275e2c714bc60318cb9bf591042e46fd8447abfc806bcc WatchSource:0}: Error finding container ae515a6cf138a51736275e2c714bc60318cb9bf591042e46fd8447abfc806bcc: Status 404 returned error can't find the container with id ae515a6cf138a51736275e2c714bc60318cb9bf591042e46fd8447abfc806bcc Sep 11 00:57:54.897586 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.897569 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-spz6s" Sep 11 00:57:54.905385 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:54.905364 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70820178_37d7_4c9e_8240_0905cdfe37ee.slice/crio-3272eb09bee47d7eaf9049070206780fd6b77d7d9975b5cee1b113d62b095736 WatchSource:0}: Error finding container 3272eb09bee47d7eaf9049070206780fd6b77d7d9975b5cee1b113d62b095736: Status 404 returned error can't find the container with id 3272eb09bee47d7eaf9049070206780fd6b77d7d9975b5cee1b113d62b095736 Sep 11 00:57:54.916294 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.916276 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h9qsj" Sep 11 00:57:54.923847 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:54.923830 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedff204d_9959_4821_9aa8_a4c33f9b07b1.slice/crio-02290c28de54ce19045967deb669c9a984aafeddd5aae7ffd86f6913ead2cd37 WatchSource:0}: Error finding container 02290c28de54ce19045967deb669c9a984aafeddd5aae7ffd86f6913ead2cd37: Status 404 returned error can't find the container with id 02290c28de54ce19045967deb669c9a984aafeddd5aae7ffd86f6913ead2cd37 Sep 11 00:57:54.932007 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.931990 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:57:54.939515 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:54.939496 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b320dd_43e9_435d_8e4f_2bca1f1233ff.slice/crio-e57f7409aec3007fe37f95b07ab062a08ea6669f1ca91b85a3e60f4ed1e87cb9 WatchSource:0}: Error finding container e57f7409aec3007fe37f95b07ab062a08ea6669f1ca91b85a3e60f4ed1e87cb9: Status 404 returned error can't find the container with id e57f7409aec3007fe37f95b07ab062a08ea6669f1ca91b85a3e60f4ed1e87cb9 Sep 11 00:57:54.959330 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.959310 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-45g5g" Sep 11 00:57:54.965264 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.965244 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" Sep 11 00:57:54.967054 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:54.967032 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3794d977_c81b_4042_ad21_bd34d7462018.slice/crio-e48cfb71bbba5432a0802210b6d45104b61b53d71037e3477a9ac01fc0e54851 WatchSource:0}: Error finding container e48cfb71bbba5432a0802210b6d45104b61b53d71037e3477a9ac01fc0e54851: Status 404 returned error can't find the container with id e48cfb71bbba5432a0802210b6d45104b61b53d71037e3477a9ac01fc0e54851 Sep 11 00:57:54.971023 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.971007 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kt8nl" Sep 11 00:57:54.973200 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:54.973182 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecf0bd8_8c32_4bcd_a586_039de22032ad.slice/crio-f5303363f12ab0ee080a7dce34ed70d26eba07a0bd1a839fabd2d4fb72537609 WatchSource:0}: Error finding container f5303363f12ab0ee080a7dce34ed70d26eba07a0bd1a839fabd2d4fb72537609: Status 404 returned error can't find the container with id f5303363f12ab0ee080a7dce34ed70d26eba07a0bd1a839fabd2d4fb72537609 Sep 11 00:57:54.973503 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.973458 2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 00:57:54.979131 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:54.979109 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5568658_3c63_4779_86b5_df6dc73de2c7.slice/crio-2f2344bc60d5e61e480caf79ca25703a37dc607f5bd17aecdc1295a3d5b070ea WatchSource:0}: Error finding container 2f2344bc60d5e61e480caf79ca25703a37dc607f5bd17aecdc1295a3d5b070ea: Status 404 returned error can't find the container with id 2f2344bc60d5e61e480caf79ca25703a37dc607f5bd17aecdc1295a3d5b070ea Sep 11 00:57:54.979991 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.979971 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bzv5j" Sep 11 00:57:54.984908 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:54.984887 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" Sep 11 00:57:54.989495 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:54.989458 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac49dda0_6ce3_42d9_83e7_1e22ffa2a096.slice/crio-15cb6bf5fc15e1f67c41c15f4b7f532ffa0d3b4502109e1397c8a2bec804afe1 WatchSource:0}: Error finding container 15cb6bf5fc15e1f67c41c15f4b7f532ffa0d3b4502109e1397c8a2bec804afe1: Status 404 returned error can't find the container with id 15cb6bf5fc15e1f67c41c15f4b7f532ffa0d3b4502109e1397c8a2bec804afe1 Sep 11 00:57:54.993419 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:57:54.993401 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d3c13b_c771_4648_bc19_cbea081e4e4c.slice/crio-64b4b330cecd7b2f75e5763cdac4bc1825dc8864156eb76c91991c07811d95a7 WatchSource:0}: Error finding container 64b4b330cecd7b2f75e5763cdac4bc1825dc8864156eb76c91991c07811d95a7: Status 404 returned error can't find the container with id 64b4b330cecd7b2f75e5763cdac4bc1825dc8864156eb76c91991c07811d95a7 Sep 11 00:57:55.129708 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:55.129510 2554 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 00:57:55.275248 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:55.275189 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-original-pull-secret\") pod \"global-pull-secret-syncer-9rbhz\" (UID: \"6038fbb3-56b0-49d0-875a-8f21cfdc9e77\") " pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:57:55.275416 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:55.275268 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-metrics-certs\") pod \"network-metrics-daemon-87kmh\" (UID: \"8f0c9d36-ae86-4625-8dd1-0c9764c655e5\") " pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:57:55.275416 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:55.275398 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 00:57:55.275547 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:55.275457 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-metrics-certs podName:8f0c9d36-ae86-4625-8dd1-0c9764c655e5 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:56.275438247 +0000 UTC m=+3.098891742 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-metrics-certs") pod "network-metrics-daemon-87kmh" (UID: "8f0c9d36-ae86-4625-8dd1-0c9764c655e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 00:57:55.275863 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:55.275841 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 00:57:55.276013 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:55.275924 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-original-pull-secret podName:6038fbb3-56b0-49d0-875a-8f21cfdc9e77 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:56.275905476 +0000 UTC m=+3.099358980 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-original-pull-secret") pod "global-pull-secret-syncer-9rbhz" (UID: "6038fbb3-56b0-49d0-875a-8f21cfdc9e77") : object "kube-system"/"original-pull-secret" not registered Sep 11 00:57:55.375582 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:55.375550 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78czw\" (UniqueName: \"kubernetes.io/projected/d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e-kube-api-access-78czw\") pod \"network-check-target-dwb7w\" (UID: \"d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e\") " pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:57:55.375745 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:55.375717 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 00:57:55.375745 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:55.375735 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 00:57:55.375866 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:55.375747 2554 projected.go:196] Error preparing data for projected volume kube-api-access-78czw for pod openshift-network-diagnostics/network-check-target-dwb7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 00:57:55.375866 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:55.375802 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e-kube-api-access-78czw podName:d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e nodeName:}" failed. No retries permitted until 2026-09-11 00:57:56.375782569 +0000 UTC m=+3.199236071 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-78czw" (UniqueName: "kubernetes.io/projected/d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e-kube-api-access-78czw") pod "network-check-target-dwb7w" (UID: "d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 00:57:55.606499 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:55.606402 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 00:52:54 +0000 UTC" deadline="2028-02-16 21:28:01.243470827 +0000 UTC" Sep 11 00:57:55.606499 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:55.606435 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12572h30m5.637040579s" Sep 11 00:57:55.674263 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:55.674236 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:57:55.674419 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:55.674363 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dwb7w" podUID="d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e" Sep 11 00:57:55.692982 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:55.692219 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qxf8s" event={"ID":"35c8be8a-e0b7-4843-b8db-4b6a486aee9e","Type":"ContainerStarted","Data":"ae515a6cf138a51736275e2c714bc60318cb9bf591042e46fd8447abfc806bcc"} Sep 11 00:57:55.698634 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:55.698605 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-45g5g" event={"ID":"3794d977-c81b-4042-ad21-bd34d7462018","Type":"ContainerStarted","Data":"e48cfb71bbba5432a0802210b6d45104b61b53d71037e3477a9ac01fc0e54851"} Sep 11 00:57:55.703351 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:55.703303 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" event={"ID":"c2b320dd-43e9-435d-8e4f-2bca1f1233ff","Type":"ContainerStarted","Data":"e57f7409aec3007fe37f95b07ab062a08ea6669f1ca91b85a3e60f4ed1e87cb9"} Sep 11 00:57:55.710431 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:55.710393 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-spz6s" event={"ID":"70820178-37d7-4c9e-8240-0905cdfe37ee","Type":"ContainerStarted","Data":"3272eb09bee47d7eaf9049070206780fd6b77d7d9975b5cee1b113d62b095736"} Sep 11 00:57:55.721773 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:55.721711 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" event={"ID":"b0d3c13b-c771-4648-bc19-cbea081e4e4c","Type":"ContainerStarted","Data":"64b4b330cecd7b2f75e5763cdac4bc1825dc8864156eb76c91991c07811d95a7"} Sep 11 00:57:55.742064 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:55.742034 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bzv5j" event={"ID":"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096","Type":"ContainerStarted","Data":"15cb6bf5fc15e1f67c41c15f4b7f532ffa0d3b4502109e1397c8a2bec804afe1"} Sep 11 00:57:55.748194 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:55.748124 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt8nl" event={"ID":"d5568658-3c63-4779-86b5-df6dc73de2c7","Type":"ContainerStarted","Data":"2f2344bc60d5e61e480caf79ca25703a37dc607f5bd17aecdc1295a3d5b070ea"} Sep 11 00:57:55.750188 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:55.750114 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" event={"ID":"7ecf0bd8-8c32-4bcd-a586-039de22032ad","Type":"ContainerStarted","Data":"f5303363f12ab0ee080a7dce34ed70d26eba07a0bd1a839fabd2d4fb72537609"} Sep 11 00:57:55.753147 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:55.753119 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h9qsj" event={"ID":"edff204d-9959-4821-9aa8-a4c33f9b07b1","Type":"ContainerStarted","Data":"02290c28de54ce19045967deb669c9a984aafeddd5aae7ffd86f6913ead2cd37"} Sep 11 00:57:55.963055 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:55.962982 2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 00:57:56.282130 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:56.282053 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-original-pull-secret\") pod \"global-pull-secret-syncer-9rbhz\" (UID: \"6038fbb3-56b0-49d0-875a-8f21cfdc9e77\") " pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:57:56.282130 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:56.282126 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-metrics-certs\") pod \"network-metrics-daemon-87kmh\" (UID: \"8f0c9d36-ae86-4625-8dd1-0c9764c655e5\") " pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:57:56.282356 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:56.282209 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 00:57:56.282356 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:56.282279 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-original-pull-secret podName:6038fbb3-56b0-49d0-875a-8f21cfdc9e77 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:58.28225885 +0000 UTC m=+5.105712355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-original-pull-secret") pod "global-pull-secret-syncer-9rbhz" (UID: "6038fbb3-56b0-49d0-875a-8f21cfdc9e77") : object "kube-system"/"original-pull-secret" not registered Sep 11 00:57:56.282356 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:56.282278 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 00:57:56.282356 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:56.282348 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-metrics-certs podName:8f0c9d36-ae86-4625-8dd1-0c9764c655e5 nodeName:}" failed. No retries permitted until 2026-09-11 00:57:58.282331407 +0000 UTC m=+5.105784918 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-metrics-certs") pod "network-metrics-daemon-87kmh" (UID: "8f0c9d36-ae86-4625-8dd1-0c9764c655e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 00:57:56.383497 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:56.383446 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78czw\" (UniqueName: \"kubernetes.io/projected/d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e-kube-api-access-78czw\") pod \"network-check-target-dwb7w\" (UID: \"d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e\") " pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:57:56.383696 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:56.383682 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 00:57:56.383755 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:56.383704 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 00:57:56.383755 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:56.383717 2554 projected.go:196] Error preparing data for projected volume kube-api-access-78czw for pod openshift-network-diagnostics/network-check-target-dwb7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 00:57:56.383873 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:56.383778 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e-kube-api-access-78czw podName:d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e nodeName:}" failed. No retries permitted until 2026-09-11 00:57:58.383760569 +0000 UTC m=+5.207214062 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-78czw" (UniqueName: "kubernetes.io/projected/d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e-kube-api-access-78czw") pod "network-check-target-dwb7w" (UID: "d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 00:57:56.564653 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:56.564580 2554 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 00:57:56.607442 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:56.607321 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 00:52:54 +0000 UTC" deadline="2028-05-18 18:03:40.67523849 +0000 UTC" Sep 11 00:57:56.607442 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:56.607372 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14777h5m44.067870139s" Sep 11 00:57:56.671995 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:56.671964 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:57:56.672131 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:56.672088 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rbhz" podUID="6038fbb3-56b0-49d0-875a-8f21cfdc9e77" Sep 11 00:57:56.672493 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:56.672474 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:57:56.672602 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:56.672583 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87kmh" podUID="8f0c9d36-ae86-4625-8dd1-0c9764c655e5" Sep 11 00:57:57.672641 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:57.671902 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:57:57.672641 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:57.672037 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dwb7w" podUID="d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e" Sep 11 00:57:58.299711 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:58.299075 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-metrics-certs\") pod \"network-metrics-daemon-87kmh\" (UID: \"8f0c9d36-ae86-4625-8dd1-0c9764c655e5\") " pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:57:58.299711 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:58.299152 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-original-pull-secret\") pod \"global-pull-secret-syncer-9rbhz\" (UID: \"6038fbb3-56b0-49d0-875a-8f21cfdc9e77\") " pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:57:58.299711 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:58.299297 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 00:57:58.299711 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:58.299332 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 00:57:58.299711 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:58.299357 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-original-pull-secret podName:6038fbb3-56b0-49d0-875a-8f21cfdc9e77 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:02.299339933 +0000 UTC m=+9.122793424 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-original-pull-secret") pod "global-pull-secret-syncer-9rbhz" (UID: "6038fbb3-56b0-49d0-875a-8f21cfdc9e77") : object "kube-system"/"original-pull-secret" not registered Sep 11 00:57:58.299711 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:58.299423 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-metrics-certs podName:8f0c9d36-ae86-4625-8dd1-0c9764c655e5 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:02.299403752 +0000 UTC m=+9.122857260 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-metrics-certs") pod "network-metrics-daemon-87kmh" (UID: "8f0c9d36-ae86-4625-8dd1-0c9764c655e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 00:57:58.400312 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:58.400276 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78czw\" (UniqueName: \"kubernetes.io/projected/d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e-kube-api-access-78czw\") pod \"network-check-target-dwb7w\" (UID: \"d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e\") " pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:57:58.400467 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:58.400443 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 00:57:58.400547 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:58.400470 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 00:57:58.400547 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:58.400485 2554 projected.go:196] Error preparing data for projected volume kube-api-access-78czw for pod openshift-network-diagnostics/network-check-target-dwb7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 00:57:58.400547 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:58.400543 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e-kube-api-access-78czw podName:d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e nodeName:}" failed. No retries permitted until 2026-09-11 00:58:02.400523356 +0000 UTC m=+9.223976859 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-78czw" (UniqueName: "kubernetes.io/projected/d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e-kube-api-access-78czw") pod "network-check-target-dwb7w" (UID: "d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 00:57:58.672209 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:58.672133 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:57:58.672355 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:58.672268 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rbhz" podUID="6038fbb3-56b0-49d0-875a-8f21cfdc9e77" Sep 11 00:57:58.672675 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:58.672646 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:57:58.673208 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:58.672756 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87kmh" podUID="8f0c9d36-ae86-4625-8dd1-0c9764c655e5" Sep 11 00:57:59.672384 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:57:59.672353 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:57:59.672561 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:57:59.672477 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dwb7w" podUID="d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e" Sep 11 00:58:00.671637 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:00.671604 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:58:00.672175 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:00.671738 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rbhz" podUID="6038fbb3-56b0-49d0-875a-8f21cfdc9e77" Sep 11 00:58:00.672175 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:00.671824 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:58:00.672175 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:00.671975 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87kmh" podUID="8f0c9d36-ae86-4625-8dd1-0c9764c655e5" Sep 11 00:58:01.672138 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:01.672105 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:58:01.672619 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:01.672229 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dwb7w" podUID="d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e" Sep 11 00:58:02.334464 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:02.334163 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-original-pull-secret\") pod \"global-pull-secret-syncer-9rbhz\" (UID: \"6038fbb3-56b0-49d0-875a-8f21cfdc9e77\") " pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:58:02.334464 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:02.334248 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-metrics-certs\") pod \"network-metrics-daemon-87kmh\" (UID: \"8f0c9d36-ae86-4625-8dd1-0c9764c655e5\") " pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:58:02.334464 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:02.334321 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 00:58:02.334464 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:02.334379 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 00:58:02.334464 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:02.334396 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-original-pull-secret podName:6038fbb3-56b0-49d0-875a-8f21cfdc9e77 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:10.334374896 +0000 UTC m=+17.157828407 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-original-pull-secret") pod "global-pull-secret-syncer-9rbhz" (UID: "6038fbb3-56b0-49d0-875a-8f21cfdc9e77") : object "kube-system"/"original-pull-secret" not registered Sep 11 00:58:02.334464 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:02.334433 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-metrics-certs podName:8f0c9d36-ae86-4625-8dd1-0c9764c655e5 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:10.334415802 +0000 UTC m=+17.157869302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-metrics-certs") pod "network-metrics-daemon-87kmh" (UID: "8f0c9d36-ae86-4625-8dd1-0c9764c655e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 00:58:02.435028 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:02.434983 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78czw\" (UniqueName: \"kubernetes.io/projected/d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e-kube-api-access-78czw\") pod \"network-check-target-dwb7w\" (UID: \"d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e\") " pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:58:02.435198 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:02.435172 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 00:58:02.435259 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:02.435203 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 00:58:02.435259 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:02.435216 2554 projected.go:196] Error preparing data for projected volume kube-api-access-78czw for pod openshift-network-diagnostics/network-check-target-dwb7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 00:58:02.435373 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:02.435283 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e-kube-api-access-78czw podName:d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e nodeName:}" failed. No retries permitted until 2026-09-11 00:58:10.435255616 +0000 UTC m=+17.258709120 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-78czw" (UniqueName: "kubernetes.io/projected/d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e-kube-api-access-78czw") pod "network-check-target-dwb7w" (UID: "d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 00:58:02.672694 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:02.672187 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:58:02.672694 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:02.672326 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87kmh" podUID="8f0c9d36-ae86-4625-8dd1-0c9764c655e5" Sep 11 00:58:02.673229 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:02.672694 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:58:02.673229 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:02.672797 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rbhz" podUID="6038fbb3-56b0-49d0-875a-8f21cfdc9e77" Sep 11 00:58:03.672456 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:03.672351 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:58:03.672607 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:03.672459 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dwb7w" podUID="d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e" Sep 11 00:58:04.671687 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:04.671655 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:58:04.672107 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:04.671656 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:58:04.672107 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:04.671773 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87kmh" podUID="8f0c9d36-ae86-4625-8dd1-0c9764c655e5" Sep 11 00:58:04.672107 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:04.671867 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rbhz" podUID="6038fbb3-56b0-49d0-875a-8f21cfdc9e77" Sep 11 00:58:05.671933 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:05.671901 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:58:05.672261 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:05.671996 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dwb7w" podUID="d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e" Sep 11 00:58:06.672338 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:06.672302 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:58:06.672338 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:06.672332 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:58:06.672796 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:06.672429 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rbhz" podUID="6038fbb3-56b0-49d0-875a-8f21cfdc9e77" Sep 11 00:58:06.672796 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:06.672528 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87kmh" podUID="8f0c9d36-ae86-4625-8dd1-0c9764c655e5" Sep 11 00:58:07.671688 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:07.671655 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:58:07.671835 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:07.671788 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dwb7w" podUID="d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e" Sep 11 00:58:08.672081 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:08.672039 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:58:08.672081 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:08.672086 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:58:08.672549 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:08.672169 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rbhz" podUID="6038fbb3-56b0-49d0-875a-8f21cfdc9e77" Sep 11 00:58:08.672549 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:08.672352 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87kmh" podUID="8f0c9d36-ae86-4625-8dd1-0c9764c655e5" Sep 11 00:58:09.672518 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:09.672481 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:58:09.673010 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:09.672612 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dwb7w" podUID="d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e" Sep 11 00:58:10.393330 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:10.393289 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-original-pull-secret\") pod \"global-pull-secret-syncer-9rbhz\" (UID: \"6038fbb3-56b0-49d0-875a-8f21cfdc9e77\") " pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:58:10.393566 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:10.393362 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-metrics-certs\") pod \"network-metrics-daemon-87kmh\" (UID: \"8f0c9d36-ae86-4625-8dd1-0c9764c655e5\") " pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:58:10.393566 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:10.393452 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 00:58:10.393566 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:10.393465 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 00:58:10.393566 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:10.393527 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-original-pull-secret podName:6038fbb3-56b0-49d0-875a-8f21cfdc9e77 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:26.393502601 +0000 UTC m=+33.216956123 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-original-pull-secret") pod "global-pull-secret-syncer-9rbhz" (UID: "6038fbb3-56b0-49d0-875a-8f21cfdc9e77") : object "kube-system"/"original-pull-secret" not registered Sep 11 00:58:10.393566 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:10.393550 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-metrics-certs podName:8f0c9d36-ae86-4625-8dd1-0c9764c655e5 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:26.393539698 +0000 UTC m=+33.216993186 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-metrics-certs") pod "network-metrics-daemon-87kmh" (UID: "8f0c9d36-ae86-4625-8dd1-0c9764c655e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 00:58:10.494118 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:10.494086 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78czw\" (UniqueName: \"kubernetes.io/projected/d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e-kube-api-access-78czw\") pod \"network-check-target-dwb7w\" (UID: \"d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e\") " pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:58:10.494277 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:10.494250 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 00:58:10.494277 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:10.494275 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 00:58:10.494370 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:10.494286 2554 projected.go:196] Error preparing data for projected volume kube-api-access-78czw for pod openshift-network-diagnostics/network-check-target-dwb7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 00:58:10.494370 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:10.494340 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e-kube-api-access-78czw podName:d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e nodeName:}" failed. No retries permitted until 2026-09-11 00:58:26.494325802 +0000 UTC m=+33.317779291 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-78czw" (UniqueName: "kubernetes.io/projected/d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e-kube-api-access-78czw") pod "network-check-target-dwb7w" (UID: "d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 00:58:10.672550 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:10.672475 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:58:10.672960 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:10.672474 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:58:10.672960 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:10.672586 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rbhz" podUID="6038fbb3-56b0-49d0-875a-8f21cfdc9e77" Sep 11 00:58:10.672960 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:10.672707 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87kmh" podUID="8f0c9d36-ae86-4625-8dd1-0c9764c655e5" Sep 11 00:58:11.671552 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:11.671523 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:58:11.671719 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:11.671644 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dwb7w" podUID="d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e" Sep 11 00:58:12.671997 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:12.671968 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:58:12.671997 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:12.671998 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:58:12.672477 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:12.672070 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rbhz" podUID="6038fbb3-56b0-49d0-875a-8f21cfdc9e77" Sep 11 00:58:12.672477 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:12.672220 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87kmh" podUID="8f0c9d36-ae86-4625-8dd1-0c9764c655e5" Sep 11 00:58:13.672531 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:13.672510 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:58:13.672799 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:13.672583 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dwb7w" podUID="d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e" Sep 11 00:58:13.801317 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:13.801241 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" event={"ID":"b0d3c13b-c771-4648-bc19-cbea081e4e4c","Type":"ContainerStarted","Data":"d9f31011a909bb301bca302343af71f65fb86ad9db02c9f7ad38b2f5289218b5"} Sep 11 00:58:13.803527 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:13.803505 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-107.ec2.internal" event={"ID":"81a39435d308b5114d55354c422903c5","Type":"ContainerStarted","Data":"803e672ba80636af65616940f8fd35056622675966c7c0a63ba78ab892b45180"} Sep 11 00:58:13.817455 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:13.817413 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-107.ec2.internal" podStartSLOduration=19.817399461 podStartE2EDuration="19.817399461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:13.817131165 +0000 UTC m=+20.640584675" watchObservedRunningTime="2026-09-11 00:58:13.817399461 +0000 UTC m=+20.640852971" Sep 11 00:58:14.671673 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:14.671508 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:58:14.671780 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:14.671508 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:58:14.671780 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:14.671746 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87kmh" podUID="8f0c9d36-ae86-4625-8dd1-0c9764c655e5" Sep 11 00:58:14.671876 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:14.671803 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rbhz" podUID="6038fbb3-56b0-49d0-875a-8f21cfdc9e77" Sep 11 00:58:14.806679 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:14.806655 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bzv5j" event={"ID":"ac49dda0-6ce3-42d9-83e7-1e22ffa2a096","Type":"ContainerStarted","Data":"ce3195c8600e1cad2e8347310d7e2963c452b38e8f809a6baa916f62b853d409"} Sep 11 00:58:14.808749 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:14.808725 2554 generic.go:356] "Generic (PLEG): container finished" podID="d5568658-3c63-4779-86b5-df6dc73de2c7" containerID="c482c697864a8500a1c31b3ed002163214881244d1bd0e83e9af6412ec9cc4d4" exitCode=0 Sep 11 00:58:14.808856 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:14.808802 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt8nl" event={"ID":"d5568658-3c63-4779-86b5-df6dc73de2c7","Type":"ContainerDied","Data":"c482c697864a8500a1c31b3ed002163214881244d1bd0e83e9af6412ec9cc4d4"} Sep 11 00:58:14.810443 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:14.810421 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" event={"ID":"7ecf0bd8-8c32-4bcd-a586-039de22032ad","Type":"ContainerStarted","Data":"654593d06fb7e049dddc97d0d6d71038e04ab09c46189d8b84bdd59f4b7ce489"} Sep 11 00:58:14.811891 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:14.811867 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qxf8s" event={"ID":"35c8be8a-e0b7-4843-b8db-4b6a486aee9e","Type":"ContainerStarted","Data":"9ddd52f7157086219a69782788d8864c2e8c49a03674db0cb73a6a663f17ad15"} Sep 11 00:58:14.813305 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:14.813282 2554 generic.go:356] "Generic (PLEG): container finished" podID="a1b1c28ec753bb4b358049fd0c5cad38" containerID="76235e7ddcd70fb12e5d2552084855ab9449d6c8ba723e27a00ec4650bc17ffe" exitCode=0 Sep 11 00:58:14.813443 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:14.813357 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-107.ec2.internal" event={"ID":"a1b1c28ec753bb4b358049fd0c5cad38","Type":"ContainerDied","Data":"76235e7ddcd70fb12e5d2552084855ab9449d6c8ba723e27a00ec4650bc17ffe"} Sep 11 00:58:14.815300 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:14.815273 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-45g5g" event={"ID":"3794d977-c81b-4042-ad21-bd34d7462018","Type":"ContainerStarted","Data":"0a28233a188f8e4663639d2c30a7db50770fb5c7df54dc86f66991ecfbd16c86"} Sep 11 00:58:14.818061 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:14.818045 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wtd8v_c2b320dd-43e9-435d-8e4f-2bca1f1233ff/ovn-acl-logging/0.log" Sep 11 00:58:14.818437 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:14.818420 2554 generic.go:356] "Generic (PLEG): container finished" podID="c2b320dd-43e9-435d-8e4f-2bca1f1233ff" containerID="717796e4437c3d604fe409fb866f85e04b2d7b34038f1b23c5ad2e612e5b3609" exitCode=1 Sep 11 00:58:14.818493 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:14.818480 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" event={"ID":"c2b320dd-43e9-435d-8e4f-2bca1f1233ff","Type":"ContainerStarted","Data":"a260f3550bbb533dc0e73f009e1f4ad11e9a57a07404c475ea1fa8554a369ad4"} Sep 11 00:58:14.818552 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:14.818538 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" event={"ID":"c2b320dd-43e9-435d-8e4f-2bca1f1233ff","Type":"ContainerStarted","Data":"2b4a2bb93b2373dc63e86ffbd3fb186c0e65a98be7a18d82b3ab7cac7bce6825"} Sep 11 00:58:14.818583 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:14.818576 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" event={"ID":"c2b320dd-43e9-435d-8e4f-2bca1f1233ff","Type":"ContainerStarted","Data":"ea08ffd0a5253cb895848506093cb4db6ac9b3db3fdedba5f8844daa2155050b"} Sep 11 00:58:14.818614 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:14.818605 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" event={"ID":"c2b320dd-43e9-435d-8e4f-2bca1f1233ff","Type":"ContainerStarted","Data":"5489154a27054ce7d74e44362ecc39529db30d303483a08728856e91cdce2617"} Sep 11 00:58:14.818654 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:14.818640 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" event={"ID":"c2b320dd-43e9-435d-8e4f-2bca1f1233ff","Type":"ContainerDied","Data":"717796e4437c3d604fe409fb866f85e04b2d7b34038f1b23c5ad2e612e5b3609"} Sep 11 00:58:14.818751 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:14.818731 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" event={"ID":"c2b320dd-43e9-435d-8e4f-2bca1f1233ff","Type":"ContainerStarted","Data":"ae233b5d217da2c11cab5cc5f927eb036d00f205d7aede7dd967493b1573da4d"} Sep 11 00:58:14.819940 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:14.819911 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-spz6s" event={"ID":"70820178-37d7-4c9e-8240-0905cdfe37ee","Type":"ContainerStarted","Data":"66c18edf6edf2dc71f31a59f127d59bf8c28ec0e36b6881a3404154d66bff1ad"} Sep 11 00:58:14.825623 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:14.825584 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bzv5j" podStartSLOduration=3.259207145 podStartE2EDuration="21.825571204s" totalImagesPullingTime="18.566364059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:14.82498172 +0000 UTC m=+21.648435232" watchObservedRunningTime="2026-09-11 00:58:14.825571204 +0000 UTC m=+21.649024714" Sep 11 00:58:14.840189 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:14.840125 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-45g5g" podStartSLOduration=3.285674548 podStartE2EDuration="21.840112956s" totalImagesPullingTime="18.554438408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:14.83961453 +0000 UTC m=+21.663068061" watchObservedRunningTime="2026-09-11 00:58:14.840112956 +0000 UTC m=+21.663566465" Sep 11 00:58:14.894338 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:14.894302 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-spz6s" podStartSLOduration=3.332493169 podStartE2EDuration="21.894290657s" totalImagesPullingTime="18.561797488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:14.892947542 +0000 UTC m=+21.716401053" watchObservedRunningTime="2026-09-11 00:58:14.894290657 +0000 UTC m=+21.717744167" Sep 11 00:58:14.919365 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:14.919325 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8tgqd" podStartSLOduration=3.389957756 podStartE2EDuration="21.919309354s" totalImagesPullingTime="18.529351598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:14.919112598 +0000 UTC m=+21.742566107" watchObservedRunningTime="2026-09-11 00:58:14.919309354 +0000 UTC m=+21.742762864" Sep 11 00:58:14.933535 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:14.933372 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qxf8s" podStartSLOduration=3.299117639 podStartE2EDuration="21.933357598s" totalImagesPullingTime="18.634239959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:14.932580039 +0000 UTC m=+21.756033550" watchObservedRunningTime="2026-09-11 00:58:14.933357598 +0000 UTC m=+21.756811113" Sep 11 00:58:15.330289 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:15.330250 2554 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 00:58:15.632796 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:15.632614 2554 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T00:58:15.33026952Z","UUID":"8c4ed9d0-b69e-4db7-a547-0376f3384c3d","Handler":null,"Name":"","Endpoint":""} Sep 11 00:58:15.634730 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:15.634671 2554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 00:58:15.634730 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:15.634708 2554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 00:58:15.671712 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:15.671680 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:58:15.671866 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:15.671793 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dwb7w" podUID="d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e" Sep 11 00:58:15.824355 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:15.824270 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-107.ec2.internal" event={"ID":"a1b1c28ec753bb4b358049fd0c5cad38","Type":"ContainerStarted","Data":"850fc5fc0ffa87d3d580d11f3d32c8fae05769be94b670c946bbcb3d071f953f"} Sep 11 00:58:15.826631 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:15.826598 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" event={"ID":"7ecf0bd8-8c32-4bcd-a586-039de22032ad","Type":"ContainerStarted","Data":"ee921ba7ba294f005e2d47eff27090ece396ffbdad92e09658e9f87994ab1902"} Sep 11 00:58:15.828637 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:15.828612 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h9qsj" event={"ID":"edff204d-9959-4821-9aa8-a4c33f9b07b1","Type":"ContainerStarted","Data":"79373da6ff00ec870b6bea50e1d68f21dd73689d3e6234cde693fda0284761f1"} Sep 11 00:58:15.838949 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:15.838898 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-107.ec2.internal" podStartSLOduration=21.838879799 podStartE2EDuration="21.838879799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:15.83826394 +0000 UTC m=+22.661717513" watchObservedRunningTime="2026-09-11 00:58:15.838879799 +0000 UTC m=+22.662333310" Sep 11 00:58:15.852122 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:15.852083 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-h9qsj" podStartSLOduration=4.287140991 podStartE2EDuration="22.852071752s" totalImagesPullingTime="18.564930761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:15.851686975 +0000 UTC m=+22.675140487" watchObservedRunningTime="2026-09-11 00:58:15.852071752 +0000 UTC m=+22.675525264" Sep 11 00:58:16.672186 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:16.672157 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:58:16.672335 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:16.672157 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:58:16.672335 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:16.672275 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rbhz" podUID="6038fbb3-56b0-49d0-875a-8f21cfdc9e77" Sep 11 00:58:16.672450 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:16.672379 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87kmh" podUID="8f0c9d36-ae86-4625-8dd1-0c9764c655e5" Sep 11 00:58:16.834708 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:16.834630 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wtd8v_c2b320dd-43e9-435d-8e4f-2bca1f1233ff/ovn-acl-logging/0.log" Sep 11 00:58:16.835150 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:16.835118 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" event={"ID":"c2b320dd-43e9-435d-8e4f-2bca1f1233ff","Type":"ContainerStarted","Data":"be7db2a1ffffe34c6070a1432c75605477c364ca24f7a33a374043004b36db67"} Sep 11 00:58:16.837564 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:16.837528 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" event={"ID":"7ecf0bd8-8c32-4bcd-a586-039de22032ad","Type":"ContainerStarted","Data":"a1a836db30effda8865718a1f1e47be54e7a1c1b48bad581e143c0d53d0e3283"} Sep 11 00:58:16.855588 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:16.855542 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d55hf" podStartSLOduration=3.715561359 podStartE2EDuration="23.855526932s" totalImagesPullingTime="20.139965573s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:16.854811082 +0000 UTC m=+23.678264592" watchObservedRunningTime="2026-09-11 00:58:16.855526932 +0000 UTC m=+23.678980441" Sep 11 00:58:17.026420 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:17.026393 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-45g5g" Sep 11 00:58:17.027131 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:17.027107 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-45g5g" Sep 11 00:58:17.672414 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:17.672385 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:58:17.672664 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:17.672505 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dwb7w" podUID="d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e" Sep 11 00:58:17.839709 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:17.839678 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-45g5g" Sep 11 00:58:17.840149 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:17.840115 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-45g5g" Sep 11 00:58:18.671681 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:18.671650 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:58:18.671681 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:18.671657 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:58:18.671889 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:18.671765 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rbhz" podUID="6038fbb3-56b0-49d0-875a-8f21cfdc9e77" Sep 11 00:58:18.671973 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:18.671925 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87kmh" podUID="8f0c9d36-ae86-4625-8dd1-0c9764c655e5" Sep 11 00:58:19.380552 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:19.380383 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:19.468287 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:19.468220 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:19.671718 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:19.671693 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:58:19.671963 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:19.671809 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dwb7w" podUID="d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e" Sep 11 00:58:19.846352 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:19.846302 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wtd8v_c2b320dd-43e9-435d-8e4f-2bca1f1233ff/ovn-acl-logging/0.log" Sep 11 00:58:19.846754 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:19.846729 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" event={"ID":"c2b320dd-43e9-435d-8e4f-2bca1f1233ff","Type":"ContainerStarted","Data":"816373b4d8dd424dc5d4df849ac6d9cb6c6f683407d8c38f702da96fddbbcf5e"} Sep 11 00:58:19.846989 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:19.846970 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:58:19.847181 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:19.847163 2554 scope.go:117] "RemoveContainer" containerID="717796e4437c3d604fe409fb866f85e04b2d7b34038f1b23c5ad2e612e5b3609" Sep 11 00:58:19.849005 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:19.848982 2554 generic.go:356] "Generic (PLEG): container finished" podID="d5568658-3c63-4779-86b5-df6dc73de2c7" containerID="268d8ddc8272a5916c1ee9619442fe3da5e39137cc0d9189a539fc345c1b473c" exitCode=0 Sep 11 00:58:19.849118 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:19.849031 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt8nl" event={"ID":"d5568658-3c63-4779-86b5-df6dc73de2c7","Type":"ContainerDied","Data":"268d8ddc8272a5916c1ee9619442fe3da5e39137cc0d9189a539fc345c1b473c"} Sep 11 00:58:19.867394 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:19.867375 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:58:20.671796 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:20.671666 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:58:20.672297 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:20.671738 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:58:20.672297 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:20.671886 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rbhz" podUID="6038fbb3-56b0-49d0-875a-8f21cfdc9e77" Sep 11 00:58:20.672297 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:20.671984 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87kmh" podUID="8f0c9d36-ae86-4625-8dd1-0c9764c655e5" Sep 11 00:58:20.753283 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:20.753252 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9rbhz"] Sep 11 00:58:20.754051 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:20.754030 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dwb7w"] Sep 11 00:58:20.754139 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:20.754132 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:58:20.754235 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:20.754210 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dwb7w" podUID="d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e" Sep 11 00:58:20.755627 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:20.755605 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-87kmh"] Sep 11 00:58:20.854768 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:20.854689 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wtd8v_c2b320dd-43e9-435d-8e4f-2bca1f1233ff/ovn-acl-logging/0.log" Sep 11 00:58:20.855083 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:20.855063 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" event={"ID":"c2b320dd-43e9-435d-8e4f-2bca1f1233ff","Type":"ContainerStarted","Data":"b8c0dbc02d2e443b5eb377588764fdc60554d403c42bbb92e2da3201c1eda455"} Sep 11 00:58:20.855387 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:20.855366 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:58:20.855476 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:20.855452 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:58:20.857237 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:20.857216 2554 generic.go:356] "Generic (PLEG): container finished" podID="d5568658-3c63-4779-86b5-df6dc73de2c7" containerID="c9d2900f64a38f2fe787731a03910e613183a3d59f27db0d51e7d15151f3d855" exitCode=0 Sep 11 00:58:20.857327 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:20.857282 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:58:20.857327 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:20.857300 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt8nl" event={"ID":"d5568658-3c63-4779-86b5-df6dc73de2c7","Type":"ContainerDied","Data":"c9d2900f64a38f2fe787731a03910e613183a3d59f27db0d51e7d15151f3d855"} Sep 11 00:58:20.857441 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:20.857428 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:58:20.857523 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:20.857506 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rbhz" podUID="6038fbb3-56b0-49d0-875a-8f21cfdc9e77" Sep 11 00:58:20.857566 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:20.857421 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87kmh" podUID="8f0c9d36-ae86-4625-8dd1-0c9764c655e5" Sep 11 00:58:20.870751 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:20.870727 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:58:20.882740 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:20.882705 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" podStartSLOduration=8.952693999 podStartE2EDuration="27.882693502s" totalImagesPullingTime="18.929999503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:20.882287804 +0000 UTC m=+27.705741315" watchObservedRunningTime="2026-09-11 00:58:20.882693502 +0000 UTC m=+27.706147012" Sep 11 00:58:21.509037 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:21.508988 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:21.563092 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:21.563052 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:21.861875 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:21.861828 2554 generic.go:356] "Generic (PLEG): container finished" podID="d5568658-3c63-4779-86b5-df6dc73de2c7" containerID="6dea208105631e2e7d2fb70e2e6590f506d8173028b5865a7cc298f7d65d8a50" exitCode=0 Sep 11 00:58:21.862229 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:21.861881 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt8nl" event={"ID":"d5568658-3c63-4779-86b5-df6dc73de2c7","Type":"ContainerDied","Data":"6dea208105631e2e7d2fb70e2e6590f506d8173028b5865a7cc298f7d65d8a50"} Sep 11 00:58:22.671714 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:22.671677 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:58:22.671897 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:22.671721 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:58:22.671897 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:22.671691 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:58:22.671897 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:22.671805 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dwb7w" podUID="d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e" Sep 11 00:58:22.672096 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:22.671925 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87kmh" podUID="8f0c9d36-ae86-4625-8dd1-0c9764c655e5" Sep 11 00:58:22.672096 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:22.672014 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rbhz" podUID="6038fbb3-56b0-49d0-875a-8f21cfdc9e77" Sep 11 00:58:24.272052 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:24.272016 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:24.373229 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:24.373193 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:24.671658 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:24.671628 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:58:24.671809 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:24.671671 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:58:24.671809 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:24.671752 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87kmh" podUID="8f0c9d36-ae86-4625-8dd1-0c9764c655e5" Sep 11 00:58:24.671942 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:24.671832 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rbhz" podUID="6038fbb3-56b0-49d0-875a-8f21cfdc9e77" Sep 11 00:58:24.671942 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:24.671877 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:58:24.672042 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:24.671979 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dwb7w" podUID="d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e" Sep 11 00:58:24.790539 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:24.790501 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:24.893126 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:24.893079 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:24.995402 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:24.995308 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:25.100897 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:25.100865 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:25.824485 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:25.824456 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:25.919682 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:25.919650 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:26.409935 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:26.409912 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-metrics-certs\") pod \"network-metrics-daemon-87kmh\" (UID: \"8f0c9d36-ae86-4625-8dd1-0c9764c655e5\") " pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:58:26.410058 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:26.409959 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-original-pull-secret\") pod \"global-pull-secret-syncer-9rbhz\" (UID: \"6038fbb3-56b0-49d0-875a-8f21cfdc9e77\") " pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:58:26.410058 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:26.410045 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 00:58:26.410058 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:26.410049 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 00:58:26.410166 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:26.410092 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-original-pull-secret podName:6038fbb3-56b0-49d0-875a-8f21cfdc9e77 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:58.410080637 +0000 UTC m=+65.233534139 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-original-pull-secret") pod "global-pull-secret-syncer-9rbhz" (UID: "6038fbb3-56b0-49d0-875a-8f21cfdc9e77") : object "kube-system"/"original-pull-secret" not registered Sep 11 00:58:26.410166 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:26.410107 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-metrics-certs podName:8f0c9d36-ae86-4625-8dd1-0c9764c655e5 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:58.410099697 +0000 UTC m=+65.233553185 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-metrics-certs") pod "network-metrics-daemon-87kmh" (UID: "8f0c9d36-ae86-4625-8dd1-0c9764c655e5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 00:58:26.510363 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:26.510337 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78czw\" (UniqueName: \"kubernetes.io/projected/d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e-kube-api-access-78czw\") pod \"network-check-target-dwb7w\" (UID: \"d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e\") " pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:58:26.510462 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:26.510448 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 00:58:26.510504 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:26.510464 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 00:58:26.510504 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:26.510472 2554 projected.go:196] Error preparing data for projected volume kube-api-access-78czw for pod openshift-network-diagnostics/network-check-target-dwb7w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 00:58:26.510572 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:26.510514 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e-kube-api-access-78czw podName:d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e nodeName:}" failed. No retries permitted until 2026-09-11 00:58:58.510499671 +0000 UTC m=+65.333953176 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-78czw" (UniqueName: "kubernetes.io/projected/d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e-kube-api-access-78czw") pod "network-check-target-dwb7w" (UID: "d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 00:58:26.671492 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:26.671468 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:58:26.671601 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:26.671508 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:58:26.671601 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:26.671475 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:58:26.671601 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:26.671574 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dwb7w" podUID="d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e" Sep 11 00:58:26.671755 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:26.671728 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87kmh" podUID="8f0c9d36-ae86-4625-8dd1-0c9764c655e5" Sep 11 00:58:26.671873 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:26.671834 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rbhz" podUID="6038fbb3-56b0-49d0-875a-8f21cfdc9e77" Sep 11 00:58:26.851536 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:26.851506 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:26.876756 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:26.876729 2554 generic.go:356] "Generic (PLEG): container finished" podID="d5568658-3c63-4779-86b5-df6dc73de2c7" containerID="c0214e4c8345239bd3e5dc0602bd594e26067185f1185cb998887a714dc6092c" exitCode=0 Sep 11 00:58:26.876874 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:26.876792 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt8nl" event={"ID":"d5568658-3c63-4779-86b5-df6dc73de2c7","Type":"ContainerDied","Data":"c0214e4c8345239bd3e5dc0602bd594e26067185f1185cb998887a714dc6092c"} Sep 11 00:58:26.913485 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:26.913408 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:27.882789 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:27.882619 2554 generic.go:356] "Generic (PLEG): container finished" podID="d5568658-3c63-4779-86b5-df6dc73de2c7" containerID="1a3d2cd337b9fd5057125cebac063f1817e64709c1393c907b434ee0258ac6bf" exitCode=0 Sep 11 00:58:27.883151 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:27.882694 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt8nl" event={"ID":"d5568658-3c63-4779-86b5-df6dc73de2c7","Type":"ContainerDied","Data":"1a3d2cd337b9fd5057125cebac063f1817e64709c1393c907b434ee0258ac6bf"} Sep 11 00:58:27.890297 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:27.890268 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-107.ec2.internal" event="NodeReady" Sep 11 00:58:27.890662 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:27.890640 2554 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 00:58:27.947411 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:27.947387 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vbxc5"] Sep 11 00:58:27.975273 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:27.975242 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ffbjw"] Sep 11 00:58:27.975372 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:27.975359 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vbxc5" Sep 11 00:58:27.978743 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:27.978725 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 00:58:27.979026 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:27.979002 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q9kdp\"" Sep 11 00:58:27.979200 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:27.979182 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 00:58:27.999513 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:27.999489 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ffbjw"] Sep 11 00:58:27.999594 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:27.999540 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vbxc5"] Sep 11 00:58:27.999594 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:27.999497 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ffbjw" Sep 11 00:58:28.002762 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.002746 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 00:58:28.002988 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.002969 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-w7vvh\"" Sep 11 00:58:28.003081 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.003065 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 00:58:28.003414 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.003401 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 00:58:28.078419 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.078391 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:28.124423 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.124402 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7czws\" (UniqueName: \"kubernetes.io/projected/1bd830e3-7d02-4637-b2b7-cf06baad6a26-kube-api-access-7czws\") pod \"dns-default-vbxc5\" (UID: \"1bd830e3-7d02-4637-b2b7-cf06baad6a26\") " pod="openshift-dns/dns-default-vbxc5" Sep 11 00:58:28.124495 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.124439 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1bd830e3-7d02-4637-b2b7-cf06baad6a26-config-volume\") pod \"dns-default-vbxc5\" (UID: \"1bd830e3-7d02-4637-b2b7-cf06baad6a26\") " pod="openshift-dns/dns-default-vbxc5" Sep 11 00:58:28.124495 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.124458 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5zwl\" (UniqueName: \"kubernetes.io/projected/a04fe26a-2690-4d67-bd7a-c19c9d610589-kube-api-access-p5zwl\") pod \"ingress-canary-ffbjw\" (UID: \"a04fe26a-2690-4d67-bd7a-c19c9d610589\") " pod="openshift-ingress-canary/ingress-canary-ffbjw" Sep 11 00:58:28.124561 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.124529 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a04fe26a-2690-4d67-bd7a-c19c9d610589-cert\") pod \"ingress-canary-ffbjw\" (UID: \"a04fe26a-2690-4d67-bd7a-c19c9d610589\") " pod="openshift-ingress-canary/ingress-canary-ffbjw" Sep 11 00:58:28.124561 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.124553 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bd830e3-7d02-4637-b2b7-cf06baad6a26-tmp-dir\") pod \"dns-default-vbxc5\" (UID: \"1bd830e3-7d02-4637-b2b7-cf06baad6a26\") " pod="openshift-dns/dns-default-vbxc5" Sep 11 00:58:28.124625 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.124576 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bd830e3-7d02-4637-b2b7-cf06baad6a26-metrics-tls\") pod \"dns-default-vbxc5\" (UID: \"1bd830e3-7d02-4637-b2b7-cf06baad6a26\") " pod="openshift-dns/dns-default-vbxc5" Sep 11 00:58:28.150733 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.150677 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:28.225714 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.225693 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1bd830e3-7d02-4637-b2b7-cf06baad6a26-config-volume\") pod \"dns-default-vbxc5\" (UID: \"1bd830e3-7d02-4637-b2b7-cf06baad6a26\") " pod="openshift-dns/dns-default-vbxc5" Sep 11 00:58:28.225793 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.225718 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5zwl\" (UniqueName: \"kubernetes.io/projected/a04fe26a-2690-4d67-bd7a-c19c9d610589-kube-api-access-p5zwl\") pod \"ingress-canary-ffbjw\" (UID: \"a04fe26a-2690-4d67-bd7a-c19c9d610589\") " pod="openshift-ingress-canary/ingress-canary-ffbjw" Sep 11 00:58:28.225793 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.225759 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a04fe26a-2690-4d67-bd7a-c19c9d610589-cert\") pod \"ingress-canary-ffbjw\" (UID: \"a04fe26a-2690-4d67-bd7a-c19c9d610589\") " pod="openshift-ingress-canary/ingress-canary-ffbjw" Sep 11 00:58:28.225793 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.225781 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bd830e3-7d02-4637-b2b7-cf06baad6a26-tmp-dir\") pod \"dns-default-vbxc5\" (UID: \"1bd830e3-7d02-4637-b2b7-cf06baad6a26\") " pod="openshift-dns/dns-default-vbxc5" Sep 11 00:58:28.225947 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.225800 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bd830e3-7d02-4637-b2b7-cf06baad6a26-metrics-tls\") pod \"dns-default-vbxc5\" (UID: \"1bd830e3-7d02-4637-b2b7-cf06baad6a26\") " pod="openshift-dns/dns-default-vbxc5" Sep 11 00:58:28.225947 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.225821 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7czws\" (UniqueName: \"kubernetes.io/projected/1bd830e3-7d02-4637-b2b7-cf06baad6a26-kube-api-access-7czws\") pod \"dns-default-vbxc5\" (UID: \"1bd830e3-7d02-4637-b2b7-cf06baad6a26\") " pod="openshift-dns/dns-default-vbxc5" Sep 11 00:58:28.226042 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:28.225963 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 00:58:28.226042 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:28.226030 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bd830e3-7d02-4637-b2b7-cf06baad6a26-metrics-tls podName:1bd830e3-7d02-4637-b2b7-cf06baad6a26 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:28.726010751 +0000 UTC m=+35.549464242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1bd830e3-7d02-4637-b2b7-cf06baad6a26-metrics-tls") pod "dns-default-vbxc5" (UID: "1bd830e3-7d02-4637-b2b7-cf06baad6a26") : secret "dns-default-metrics-tls" not found Sep 11 00:58:28.226150 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.226120 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bd830e3-7d02-4637-b2b7-cf06baad6a26-tmp-dir\") pod \"dns-default-vbxc5\" (UID: \"1bd830e3-7d02-4637-b2b7-cf06baad6a26\") " pod="openshift-dns/dns-default-vbxc5" Sep 11 00:58:28.226264 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.226240 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1bd830e3-7d02-4637-b2b7-cf06baad6a26-config-volume\") pod \"dns-default-vbxc5\" (UID: \"1bd830e3-7d02-4637-b2b7-cf06baad6a26\") " pod="openshift-dns/dns-default-vbxc5" Sep 11 00:58:28.229866 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.229828 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a04fe26a-2690-4d67-bd7a-c19c9d610589-cert\") pod \"ingress-canary-ffbjw\" (UID: \"a04fe26a-2690-4d67-bd7a-c19c9d610589\") " pod="openshift-ingress-canary/ingress-canary-ffbjw" Sep 11 00:58:28.235783 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.235756 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7czws\" (UniqueName: \"kubernetes.io/projected/1bd830e3-7d02-4637-b2b7-cf06baad6a26-kube-api-access-7czws\") pod \"dns-default-vbxc5\" (UID: \"1bd830e3-7d02-4637-b2b7-cf06baad6a26\") " pod="openshift-dns/dns-default-vbxc5" Sep 11 00:58:28.235929 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.235908 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5zwl\" (UniqueName: \"kubernetes.io/projected/a04fe26a-2690-4d67-bd7a-c19c9d610589-kube-api-access-p5zwl\") pod \"ingress-canary-ffbjw\" (UID: \"a04fe26a-2690-4d67-bd7a-c19c9d610589\") " pod="openshift-ingress-canary/ingress-canary-ffbjw" Sep 11 00:58:28.312924 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.312904 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ffbjw" Sep 11 00:58:28.633031 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.633002 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ffbjw"] Sep 11 00:58:28.637699 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:58:28.637676 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda04fe26a_2690_4d67_bd7a_c19c9d610589.slice/crio-ac8715e48d8baa56779ec09cad568b48f9fabc81396e1d4a88d78555c49a92cf WatchSource:0}: Error finding container ac8715e48d8baa56779ec09cad568b48f9fabc81396e1d4a88d78555c49a92cf: Status 404 returned error can't find the container with id ac8715e48d8baa56779ec09cad568b48f9fabc81396e1d4a88d78555c49a92cf Sep 11 00:58:28.671523 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.671506 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:58:28.671599 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.671539 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:58:28.671599 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.671514 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:58:28.674398 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.674381 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 00:58:28.674560 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.674536 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lb8gv\"" Sep 11 00:58:28.674654 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.674563 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 00:58:28.674654 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.674635 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 00:58:28.675485 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.675470 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 00:58:28.675574 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.675503 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jwmm5\"" Sep 11 00:58:28.729754 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.729732 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bd830e3-7d02-4637-b2b7-cf06baad6a26-metrics-tls\") pod \"dns-default-vbxc5\" (UID: \"1bd830e3-7d02-4637-b2b7-cf06baad6a26\") " pod="openshift-dns/dns-default-vbxc5" Sep 11 00:58:28.729877 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:28.729864 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 00:58:28.729923 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:28.729911 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bd830e3-7d02-4637-b2b7-cf06baad6a26-metrics-tls podName:1bd830e3-7d02-4637-b2b7-cf06baad6a26 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:29.729898799 +0000 UTC m=+36.553352287 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1bd830e3-7d02-4637-b2b7-cf06baad6a26-metrics-tls") pod "dns-default-vbxc5" (UID: "1bd830e3-7d02-4637-b2b7-cf06baad6a26") : secret "dns-default-metrics-tls" not found Sep 11 00:58:28.886474 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.886446 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ffbjw" event={"ID":"a04fe26a-2690-4d67-bd7a-c19c9d610589","Type":"ContainerStarted","Data":"ac8715e48d8baa56779ec09cad568b48f9fabc81396e1d4a88d78555c49a92cf"} Sep 11 00:58:28.889997 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.889976 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kt8nl" event={"ID":"d5568658-3c63-4779-86b5-df6dc73de2c7","Type":"ContainerStarted","Data":"9840772d30e3e5b486e6d5ddd5870db3ffcc759cd92867561411a3aab1de2dff"} Sep 11 00:58:28.914487 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:28.912672 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kt8nl" podStartSLOduration=7.3164459 podStartE2EDuration="35.91265491s" totalImagesPullingTime="28.59620901s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:28.911981001 +0000 UTC m=+35.735434512" watchObservedRunningTime="2026-09-11 00:58:28.91265491 +0000 UTC m=+35.736108420" Sep 11 00:58:29.033824 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:29.033760 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:29.102030 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:29.102007 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:29.736178 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:29.736147 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bd830e3-7d02-4637-b2b7-cf06baad6a26-metrics-tls\") pod \"dns-default-vbxc5\" (UID: \"1bd830e3-7d02-4637-b2b7-cf06baad6a26\") " pod="openshift-dns/dns-default-vbxc5" Sep 11 00:58:29.736334 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:29.736298 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 00:58:29.736382 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:29.736362 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bd830e3-7d02-4637-b2b7-cf06baad6a26-metrics-tls podName:1bd830e3-7d02-4637-b2b7-cf06baad6a26 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:31.736342736 +0000 UTC m=+38.559796238 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1bd830e3-7d02-4637-b2b7-cf06baad6a26-metrics-tls") pod "dns-default-vbxc5" (UID: "1bd830e3-7d02-4637-b2b7-cf06baad6a26") : secret "dns-default-metrics-tls" not found Sep 11 00:58:29.796755 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:29.796714 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:31.749752 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:31.749723 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bd830e3-7d02-4637-b2b7-cf06baad6a26-metrics-tls\") pod \"dns-default-vbxc5\" (UID: \"1bd830e3-7d02-4637-b2b7-cf06baad6a26\") " pod="openshift-dns/dns-default-vbxc5" Sep 11 00:58:31.750113 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:31.749881 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 00:58:31.750113 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:31.749940 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bd830e3-7d02-4637-b2b7-cf06baad6a26-metrics-tls podName:1bd830e3-7d02-4637-b2b7-cf06baad6a26 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:35.749922693 +0000 UTC m=+42.573376198 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1bd830e3-7d02-4637-b2b7-cf06baad6a26-metrics-tls") pod "dns-default-vbxc5" (UID: "1bd830e3-7d02-4637-b2b7-cf06baad6a26") : secret "dns-default-metrics-tls" not found Sep 11 00:58:31.899460 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:31.899433 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ffbjw" event={"ID":"a04fe26a-2690-4d67-bd7a-c19c9d610589","Type":"ContainerStarted","Data":"a5e7e609b7cd4d5cc80141d8eb099f8c994090e3276a42456388b6b219ff525c"} Sep 11 00:58:31.915108 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:31.915068 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ffbjw" podStartSLOduration=2.213524119 podStartE2EDuration="4.915053921s" totalImagesPullingTime="2.701529802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:31.914101115 +0000 UTC m=+38.737554626" watchObservedRunningTime="2026-09-11 00:58:31.915053921 +0000 UTC m=+38.738507430" Sep 11 00:58:32.221866 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:32.221825 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:32.282478 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:32.282439 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:35.771669 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:35.771627 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bd830e3-7d02-4637-b2b7-cf06baad6a26-metrics-tls\") pod \"dns-default-vbxc5\" (UID: \"1bd830e3-7d02-4637-b2b7-cf06baad6a26\") " pod="openshift-dns/dns-default-vbxc5" Sep 11 00:58:35.772168 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:35.771783 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 00:58:35.772168 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:35.771864 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bd830e3-7d02-4637-b2b7-cf06baad6a26-metrics-tls podName:1bd830e3-7d02-4637-b2b7-cf06baad6a26 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:43.771828753 +0000 UTC m=+50.595282252 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1bd830e3-7d02-4637-b2b7-cf06baad6a26-metrics-tls") pod "dns-default-vbxc5" (UID: "1bd830e3-7d02-4637-b2b7-cf06baad6a26") : secret "dns-default-metrics-tls" not found Sep 11 00:58:37.324085 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.324049 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5n9qs"] Sep 11 00:58:37.352076 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.352052 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-tqxxh"] Sep 11 00:58:37.352196 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.352184 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5n9qs" Sep 11 00:58:37.354978 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.354958 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5hvq7\"" Sep 11 00:58:37.355398 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.355381 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 11 00:58:37.355458 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.355446 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 11 00:58:37.370065 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.370038 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-tqxxh"] Sep 11 00:58:37.370159 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.370076 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5n9qs"] Sep 11 00:58:37.370212 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.370174 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tqxxh" Sep 11 00:58:37.373169 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.373149 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 11 00:58:37.373288 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.373271 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pnh88\"" Sep 11 00:58:37.374351 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.374329 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 00:58:37.374433 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.374376 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 11 00:58:37.374481 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.374376 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 00:58:37.429360 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.429332 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mng4p"] Sep 11 00:58:37.453918 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.453894 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5946c9f774-tk9pg"] Sep 11 00:58:37.454034 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.454018 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mng4p" Sep 11 00:58:37.456617 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.456593 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 11 00:58:37.456708 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.456616 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 11 00:58:37.457163 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.457136 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-w64sj\"" Sep 11 00:58:37.457393 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.457370 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 11 00:58:37.457581 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.457496 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 11 00:58:37.478045 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.478027 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mng4p"] Sep 11 00:58:37.478148 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.478052 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5946c9f774-tk9pg"] Sep 11 00:58:37.478148 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.478116 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:37.480362 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.480348 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 11 00:58:37.480596 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.480572 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 00:58:37.480699 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.480607 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 11 00:58:37.480699 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.480613 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-nhkb9\"" Sep 11 00:58:37.480699 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.480633 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 11 00:58:37.480873 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.480836 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 11 00:58:37.480942 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.480904 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 00:58:37.483203 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.483183 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b619e4ba-cf0c-4078-b918-3dbd3f67c119-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-tqxxh\" (UID: \"b619e4ba-cf0c-4078-b918-3dbd3f67c119\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tqxxh" Sep 11 00:58:37.483341 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.483228 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjdnh\" (UniqueName: \"kubernetes.io/projected/9ba1ed98-f643-461d-92df-c6319ad47f11-kube-api-access-wjdnh\") pod \"volume-data-source-validator-86695b7cc7-5n9qs\" (UID: \"9ba1ed98-f643-461d-92df-c6319ad47f11\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5n9qs" Sep 11 00:58:37.483341 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.483252 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b619e4ba-cf0c-4078-b918-3dbd3f67c119-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-tqxxh\" (UID: \"b619e4ba-cf0c-4078-b918-3dbd3f67c119\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tqxxh" Sep 11 00:58:37.483341 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.483292 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7wpf\" (UniqueName: \"kubernetes.io/projected/b619e4ba-cf0c-4078-b918-3dbd3f67c119-kube-api-access-n7wpf\") pod \"cluster-monitoring-operator-6cd954b897-tqxxh\" (UID: \"b619e4ba-cf0c-4078-b918-3dbd3f67c119\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tqxxh" Sep 11 00:58:37.500538 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.500516 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:37.526184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.526163 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-r7p6n"] Sep 11 00:58:37.544023 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.543996 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-r7p6n"] Sep 11 00:58:37.544116 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.544051 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r7p6n" Sep 11 00:58:37.546447 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.546431 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-cd5bm\"" Sep 11 00:58:37.546447 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.546442 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 11 00:58:37.546580 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.546436 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 11 00:58:37.571501 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.571479 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:37.584467 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.584428 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dee19a50-296e-4a89-9e02-21ee8aa3ca87-default-certificate\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:37.584467 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.584452 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2145b039-e727-4271-ad8e-e15b459b2a24-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-mng4p\" (UID: \"2145b039-e727-4271-ad8e-e15b459b2a24\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mng4p" Sep 11 00:58:37.584566 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.584485 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b619e4ba-cf0c-4078-b918-3dbd3f67c119-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-tqxxh\" (UID: \"b619e4ba-cf0c-4078-b918-3dbd3f67c119\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tqxxh" Sep 11 00:58:37.584566 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.584502 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p22v\" (UniqueName: \"kubernetes.io/projected/dee19a50-296e-4a89-9e02-21ee8aa3ca87-kube-api-access-4p22v\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:37.584566 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.584533 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb2lc\" (UniqueName: \"kubernetes.io/projected/2145b039-e727-4271-ad8e-e15b459b2a24-kube-api-access-wb2lc\") pod \"kube-storage-version-migrator-operator-6c8f84f897-mng4p\" (UID: \"2145b039-e727-4271-ad8e-e15b459b2a24\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mng4p" Sep 11 00:58:37.584673 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.584580 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjdnh\" (UniqueName: \"kubernetes.io/projected/9ba1ed98-f643-461d-92df-c6319ad47f11-kube-api-access-wjdnh\") pod \"volume-data-source-validator-86695b7cc7-5n9qs\" (UID: \"9ba1ed98-f643-461d-92df-c6319ad47f11\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5n9qs" Sep 11 00:58:37.584673 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.584612 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dee19a50-296e-4a89-9e02-21ee8aa3ca87-service-ca-bundle\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:37.584673 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.584629 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dee19a50-296e-4a89-9e02-21ee8aa3ca87-stats-auth\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:37.584673 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.584649 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b619e4ba-cf0c-4078-b918-3dbd3f67c119-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-tqxxh\" (UID: \"b619e4ba-cf0c-4078-b918-3dbd3f67c119\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tqxxh" Sep 11 00:58:37.584673 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.584665 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2145b039-e727-4271-ad8e-e15b459b2a24-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-mng4p\" (UID: \"2145b039-e727-4271-ad8e-e15b459b2a24\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mng4p" Sep 11 00:58:37.584870 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.584680 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee19a50-296e-4a89-9e02-21ee8aa3ca87-metrics-certs\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:37.584870 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.584707 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7wpf\" (UniqueName: \"kubernetes.io/projected/b619e4ba-cf0c-4078-b918-3dbd3f67c119-kube-api-access-n7wpf\") pod \"cluster-monitoring-operator-6cd954b897-tqxxh\" (UID: \"b619e4ba-cf0c-4078-b918-3dbd3f67c119\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tqxxh" Sep 11 00:58:37.584870 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:37.584739 2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 00:58:37.584870 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:37.584810 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b619e4ba-cf0c-4078-b918-3dbd3f67c119-cluster-monitoring-operator-tls podName:b619e4ba-cf0c-4078-b918-3dbd3f67c119 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:38.08478441 +0000 UTC m=+44.908237901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b619e4ba-cf0c-4078-b918-3dbd3f67c119-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-tqxxh" (UID: "b619e4ba-cf0c-4078-b918-3dbd3f67c119") : secret "cluster-monitoring-operator-tls" not found Sep 11 00:58:37.585114 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.585098 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b619e4ba-cf0c-4078-b918-3dbd3f67c119-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-tqxxh\" (UID: \"b619e4ba-cf0c-4078-b918-3dbd3f67c119\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tqxxh" Sep 11 00:58:37.594106 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.594082 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjdnh\" (UniqueName: \"kubernetes.io/projected/9ba1ed98-f643-461d-92df-c6319ad47f11-kube-api-access-wjdnh\") pod \"volume-data-source-validator-86695b7cc7-5n9qs\" (UID: \"9ba1ed98-f643-461d-92df-c6319ad47f11\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5n9qs" Sep 11 00:58:37.594178 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.594086 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7wpf\" (UniqueName: \"kubernetes.io/projected/b619e4ba-cf0c-4078-b918-3dbd3f67c119-kube-api-access-n7wpf\") pod \"cluster-monitoring-operator-6cd954b897-tqxxh\" (UID: \"b619e4ba-cf0c-4078-b918-3dbd3f67c119\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tqxxh" Sep 11 00:58:37.632205 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.632182 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:37.661319 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.661298 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5n9qs" Sep 11 00:58:37.685729 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.685709 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dee19a50-296e-4a89-9e02-21ee8aa3ca87-service-ca-bundle\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:37.685821 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.685736 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dee19a50-296e-4a89-9e02-21ee8aa3ca87-stats-auth\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:37.685821 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.685769 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2145b039-e727-4271-ad8e-e15b459b2a24-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-mng4p\" (UID: \"2145b039-e727-4271-ad8e-e15b459b2a24\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mng4p" Sep 11 00:58:37.685821 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.685793 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee19a50-296e-4a89-9e02-21ee8aa3ca87-metrics-certs\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:37.686042 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.685824 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/746635c6-8538-4d95-ad45-7f573caee4b3-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-r7p6n\" (UID: \"746635c6-8538-4d95-ad45-7f573caee4b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r7p6n" Sep 11 00:58:37.686042 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.685863 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dee19a50-296e-4a89-9e02-21ee8aa3ca87-default-certificate\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:37.686042 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:37.685935 2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 00:58:37.686042 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:37.685943 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dee19a50-296e-4a89-9e02-21ee8aa3ca87-service-ca-bundle podName:dee19a50-296e-4a89-9e02-21ee8aa3ca87 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:38.185923321 +0000 UTC m=+45.009376815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dee19a50-296e-4a89-9e02-21ee8aa3ca87-service-ca-bundle") pod "router-default-5946c9f774-tk9pg" (UID: "dee19a50-296e-4a89-9e02-21ee8aa3ca87") : configmap references non-existent config key: service-ca.crt Sep 11 00:58:37.686042 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:37.686007 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee19a50-296e-4a89-9e02-21ee8aa3ca87-metrics-certs podName:dee19a50-296e-4a89-9e02-21ee8aa3ca87 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:38.185987404 +0000 UTC m=+45.009440902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dee19a50-296e-4a89-9e02-21ee8aa3ca87-metrics-certs") pod "router-default-5946c9f774-tk9pg" (UID: "dee19a50-296e-4a89-9e02-21ee8aa3ca87") : secret "router-metrics-certs-default" not found Sep 11 00:58:37.686042 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.686036 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2145b039-e727-4271-ad8e-e15b459b2a24-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-mng4p\" (UID: \"2145b039-e727-4271-ad8e-e15b459b2a24\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mng4p" Sep 11 00:58:37.686375 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.686090 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p22v\" (UniqueName: \"kubernetes.io/projected/dee19a50-296e-4a89-9e02-21ee8aa3ca87-kube-api-access-4p22v\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:37.686375 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.686119 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb2lc\" (UniqueName: \"kubernetes.io/projected/2145b039-e727-4271-ad8e-e15b459b2a24-kube-api-access-wb2lc\") pod \"kube-storage-version-migrator-operator-6c8f84f897-mng4p\" (UID: \"2145b039-e727-4271-ad8e-e15b459b2a24\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mng4p" Sep 11 00:58:37.686375 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.686163 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/746635c6-8538-4d95-ad45-7f573caee4b3-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-r7p6n\" (UID: \"746635c6-8538-4d95-ad45-7f573caee4b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r7p6n" Sep 11 00:58:37.686528 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.686410 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2145b039-e727-4271-ad8e-e15b459b2a24-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-mng4p\" (UID: \"2145b039-e727-4271-ad8e-e15b459b2a24\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mng4p" Sep 11 00:58:37.688152 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.688125 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2145b039-e727-4271-ad8e-e15b459b2a24-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-mng4p\" (UID: \"2145b039-e727-4271-ad8e-e15b459b2a24\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mng4p" Sep 11 00:58:37.688331 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.688314 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dee19a50-296e-4a89-9e02-21ee8aa3ca87-stats-auth\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:37.688420 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.688402 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dee19a50-296e-4a89-9e02-21ee8aa3ca87-default-certificate\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:37.694143 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.694118 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb2lc\" (UniqueName: \"kubernetes.io/projected/2145b039-e727-4271-ad8e-e15b459b2a24-kube-api-access-wb2lc\") pod \"kube-storage-version-migrator-operator-6c8f84f897-mng4p\" (UID: \"2145b039-e727-4271-ad8e-e15b459b2a24\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mng4p" Sep 11 00:58:37.694323 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.694304 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p22v\" (UniqueName: \"kubernetes.io/projected/dee19a50-296e-4a89-9e02-21ee8aa3ca87-kube-api-access-4p22v\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:37.763102 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.763075 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mng4p" Sep 11 00:58:37.787440 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.787421 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/746635c6-8538-4d95-ad45-7f573caee4b3-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-r7p6n\" (UID: \"746635c6-8538-4d95-ad45-7f573caee4b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r7p6n" Sep 11 00:58:37.787519 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.787476 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/746635c6-8538-4d95-ad45-7f573caee4b3-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-r7p6n\" (UID: \"746635c6-8538-4d95-ad45-7f573caee4b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r7p6n" Sep 11 00:58:37.787591 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:37.787572 2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 00:58:37.787644 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:37.787636 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/746635c6-8538-4d95-ad45-7f573caee4b3-networking-console-plugin-cert podName:746635c6-8538-4d95-ad45-7f573caee4b3 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:38.287621006 +0000 UTC m=+45.111074494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/746635c6-8538-4d95-ad45-7f573caee4b3-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-r7p6n" (UID: "746635c6-8538-4d95-ad45-7f573caee4b3") : secret "networking-console-plugin-cert" not found Sep 11 00:58:37.788075 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.788059 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/746635c6-8538-4d95-ad45-7f573caee4b3-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-r7p6n\" (UID: \"746635c6-8538-4d95-ad45-7f573caee4b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r7p6n" Sep 11 00:58:37.946033 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:37.945994 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5n9qs"] Sep 11 00:58:37.948414 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:58:37.948391 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba1ed98_f643_461d_92df_c6319ad47f11.slice/crio-9598a215294def4cd2807f4703b9956bc930ff0ea6656a62d763b3ccb63e46bf WatchSource:0}: Error finding container 9598a215294def4cd2807f4703b9956bc930ff0ea6656a62d763b3ccb63e46bf: Status 404 returned error can't find the container with id 9598a215294def4cd2807f4703b9956bc930ff0ea6656a62d763b3ccb63e46bf Sep 11 00:58:38.056180 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:38.056157 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mng4p"] Sep 11 00:58:38.058587 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:58:38.058564 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2145b039_e727_4271_ad8e_e15b459b2a24.slice/crio-989c2db00029e005aff430153b57f1403b661bbb3851b6805f07fc5fdd1e5d08 WatchSource:0}: Error finding container 989c2db00029e005aff430153b57f1403b661bbb3851b6805f07fc5fdd1e5d08: Status 404 returned error can't find the container with id 989c2db00029e005aff430153b57f1403b661bbb3851b6805f07fc5fdd1e5d08 Sep 11 00:58:38.089681 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:38.089634 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b619e4ba-cf0c-4078-b918-3dbd3f67c119-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-tqxxh\" (UID: \"b619e4ba-cf0c-4078-b918-3dbd3f67c119\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tqxxh" Sep 11 00:58:38.089784 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:38.089766 2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 00:58:38.089860 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:38.089832 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b619e4ba-cf0c-4078-b918-3dbd3f67c119-cluster-monitoring-operator-tls podName:b619e4ba-cf0c-4078-b918-3dbd3f67c119 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:39.089817067 +0000 UTC m=+45.913270571 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b619e4ba-cf0c-4078-b918-3dbd3f67c119-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-tqxxh" (UID: "b619e4ba-cf0c-4078-b918-3dbd3f67c119") : secret "cluster-monitoring-operator-tls" not found Sep 11 00:58:38.190482 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:38.190463 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dee19a50-296e-4a89-9e02-21ee8aa3ca87-service-ca-bundle\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:38.190557 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:38.190496 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee19a50-296e-4a89-9e02-21ee8aa3ca87-metrics-certs\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:38.190628 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:38.190615 2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 00:58:38.190665 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:38.190642 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dee19a50-296e-4a89-9e02-21ee8aa3ca87-service-ca-bundle podName:dee19a50-296e-4a89-9e02-21ee8aa3ca87 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:39.19062516 +0000 UTC m=+46.014078654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dee19a50-296e-4a89-9e02-21ee8aa3ca87-service-ca-bundle") pod "router-default-5946c9f774-tk9pg" (UID: "dee19a50-296e-4a89-9e02-21ee8aa3ca87") : configmap references non-existent config key: service-ca.crt Sep 11 00:58:38.190665 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:38.190662 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee19a50-296e-4a89-9e02-21ee8aa3ca87-metrics-certs podName:dee19a50-296e-4a89-9e02-21ee8aa3ca87 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:39.190653194 +0000 UTC m=+46.014106684 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dee19a50-296e-4a89-9e02-21ee8aa3ca87-metrics-certs") pod "router-default-5946c9f774-tk9pg" (UID: "dee19a50-296e-4a89-9e02-21ee8aa3ca87") : secret "router-metrics-certs-default" not found Sep 11 00:58:38.290798 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:38.290778 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/746635c6-8538-4d95-ad45-7f573caee4b3-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-r7p6n\" (UID: \"746635c6-8538-4d95-ad45-7f573caee4b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r7p6n" Sep 11 00:58:38.290947 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:38.290934 2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 00:58:38.290995 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:38.290982 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/746635c6-8538-4d95-ad45-7f573caee4b3-networking-console-plugin-cert podName:746635c6-8538-4d95-ad45-7f573caee4b3 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:39.290968223 +0000 UTC m=+46.114421713 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/746635c6-8538-4d95-ad45-7f573caee4b3-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-r7p6n" (UID: "746635c6-8538-4d95-ad45-7f573caee4b3") : secret "networking-console-plugin-cert" not found Sep 11 00:58:38.921774 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:38.921736 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5n9qs" event={"ID":"9ba1ed98-f643-461d-92df-c6319ad47f11","Type":"ContainerStarted","Data":"9598a215294def4cd2807f4703b9956bc930ff0ea6656a62d763b3ccb63e46bf"} Sep 11 00:58:38.923274 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:38.923234 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mng4p" event={"ID":"2145b039-e727-4271-ad8e-e15b459b2a24","Type":"ContainerStarted","Data":"989c2db00029e005aff430153b57f1403b661bbb3851b6805f07fc5fdd1e5d08"} Sep 11 00:58:39.098761 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:39.098695 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b619e4ba-cf0c-4078-b918-3dbd3f67c119-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-tqxxh\" (UID: \"b619e4ba-cf0c-4078-b918-3dbd3f67c119\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tqxxh" Sep 11 00:58:39.098896 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:39.098856 2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 00:58:39.098943 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:39.098914 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b619e4ba-cf0c-4078-b918-3dbd3f67c119-cluster-monitoring-operator-tls podName:b619e4ba-cf0c-4078-b918-3dbd3f67c119 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:41.098899622 +0000 UTC m=+47.922353127 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b619e4ba-cf0c-4078-b918-3dbd3f67c119-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-tqxxh" (UID: "b619e4ba-cf0c-4078-b918-3dbd3f67c119") : secret "cluster-monitoring-operator-tls" not found Sep 11 00:58:39.199157 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:39.199128 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dee19a50-296e-4a89-9e02-21ee8aa3ca87-service-ca-bundle\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:39.199296 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:39.199167 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee19a50-296e-4a89-9e02-21ee8aa3ca87-metrics-certs\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:39.199373 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:39.199312 2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 00:58:39.199373 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:39.199328 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dee19a50-296e-4a89-9e02-21ee8aa3ca87-service-ca-bundle podName:dee19a50-296e-4a89-9e02-21ee8aa3ca87 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:41.199304717 +0000 UTC m=+48.022758220 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dee19a50-296e-4a89-9e02-21ee8aa3ca87-service-ca-bundle") pod "router-default-5946c9f774-tk9pg" (UID: "dee19a50-296e-4a89-9e02-21ee8aa3ca87") : configmap references non-existent config key: service-ca.crt Sep 11 00:58:39.199373 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:39.199359 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee19a50-296e-4a89-9e02-21ee8aa3ca87-metrics-certs podName:dee19a50-296e-4a89-9e02-21ee8aa3ca87 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:41.199342927 +0000 UTC m=+48.022796431 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dee19a50-296e-4a89-9e02-21ee8aa3ca87-metrics-certs") pod "router-default-5946c9f774-tk9pg" (UID: "dee19a50-296e-4a89-9e02-21ee8aa3ca87") : secret "router-metrics-certs-default" not found Sep 11 00:58:39.299976 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:39.299937 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/746635c6-8538-4d95-ad45-7f573caee4b3-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-r7p6n\" (UID: \"746635c6-8538-4d95-ad45-7f573caee4b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r7p6n" Sep 11 00:58:39.300119 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:39.300054 2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 00:58:39.300119 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:39.300112 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/746635c6-8538-4d95-ad45-7f573caee4b3-networking-console-plugin-cert podName:746635c6-8538-4d95-ad45-7f573caee4b3 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:41.300096271 +0000 UTC m=+48.123549760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/746635c6-8538-4d95-ad45-7f573caee4b3-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-r7p6n" (UID: "746635c6-8538-4d95-ad45-7f573caee4b3") : secret "networking-console-plugin-cert" not found Sep 11 00:58:40.929396 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:40.929360 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mng4p" event={"ID":"2145b039-e727-4271-ad8e-e15b459b2a24","Type":"ContainerStarted","Data":"5afec9c443623a559ce6b652565870066f6d618b8c255e9d1e9e9280cec5f106"} Sep 11 00:58:40.930901 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:40.930880 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5n9qs" event={"ID":"9ba1ed98-f643-461d-92df-c6319ad47f11","Type":"ContainerStarted","Data":"18cce73ea0f33c9810480e0e66a29db9a4d53af18e1ab582ca9e65c0acbe2a61"} Sep 11 00:58:40.945537 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:40.945492 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mng4p" podStartSLOduration=1.720910866 podStartE2EDuration="3.945480774s" totalImagesPullingTime="2.224569908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:40.944567067 +0000 UTC m=+47.768020577" watchObservedRunningTime="2026-09-11 00:58:40.945480774 +0000 UTC m=+47.768934299" Sep 11 00:58:40.959781 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:40.959731 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5n9qs" podStartSLOduration=1.97633641 podStartE2EDuration="3.959715112s" totalImagesPullingTime="1.983378702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:40.958371552 +0000 UTC m=+47.781825063" watchObservedRunningTime="2026-09-11 00:58:40.959715112 +0000 UTC m=+47.783168625" Sep 11 00:58:41.038612 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:41.038584 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:41.096951 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:41.096908 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:41.114765 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:41.114740 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b619e4ba-cf0c-4078-b918-3dbd3f67c119-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-tqxxh\" (UID: \"b619e4ba-cf0c-4078-b918-3dbd3f67c119\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tqxxh" Sep 11 00:58:41.114926 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:41.114910 2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 00:58:41.114993 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:41.114980 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b619e4ba-cf0c-4078-b918-3dbd3f67c119-cluster-monitoring-operator-tls podName:b619e4ba-cf0c-4078-b918-3dbd3f67c119 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:45.114961069 +0000 UTC m=+51.938414561 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b619e4ba-cf0c-4078-b918-3dbd3f67c119-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-tqxxh" (UID: "b619e4ba-cf0c-4078-b918-3dbd3f67c119") : secret "cluster-monitoring-operator-tls" not found Sep 11 00:58:41.215701 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:41.215621 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dee19a50-296e-4a89-9e02-21ee8aa3ca87-service-ca-bundle\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:41.215701 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:41.215675 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee19a50-296e-4a89-9e02-21ee8aa3ca87-metrics-certs\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:41.215883 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:41.215776 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dee19a50-296e-4a89-9e02-21ee8aa3ca87-service-ca-bundle podName:dee19a50-296e-4a89-9e02-21ee8aa3ca87 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:45.215759653 +0000 UTC m=+52.039213159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dee19a50-296e-4a89-9e02-21ee8aa3ca87-service-ca-bundle") pod "router-default-5946c9f774-tk9pg" (UID: "dee19a50-296e-4a89-9e02-21ee8aa3ca87") : configmap references non-existent config key: service-ca.crt Sep 11 00:58:41.215883 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:41.215814 2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 00:58:41.215984 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:41.215891 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee19a50-296e-4a89-9e02-21ee8aa3ca87-metrics-certs podName:dee19a50-296e-4a89-9e02-21ee8aa3ca87 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:45.215875493 +0000 UTC m=+52.039328982 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dee19a50-296e-4a89-9e02-21ee8aa3ca87-metrics-certs") pod "router-default-5946c9f774-tk9pg" (UID: "dee19a50-296e-4a89-9e02-21ee8aa3ca87") : secret "router-metrics-certs-default" not found Sep 11 00:58:41.316123 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:41.316097 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/746635c6-8538-4d95-ad45-7f573caee4b3-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-r7p6n\" (UID: \"746635c6-8538-4d95-ad45-7f573caee4b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r7p6n" Sep 11 00:58:41.316254 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:41.316237 2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 00:58:41.316319 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:41.316297 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/746635c6-8538-4d95-ad45-7f573caee4b3-networking-console-plugin-cert podName:746635c6-8538-4d95-ad45-7f573caee4b3 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:45.316282123 +0000 UTC m=+52.139735613 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/746635c6-8538-4d95-ad45-7f573caee4b3-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-r7p6n" (UID: "746635c6-8538-4d95-ad45-7f573caee4b3") : secret "networking-console-plugin-cert" not found Sep 11 00:58:41.556883 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:41.556792 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:41.701957 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:41.701929 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:41.797001 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:41.796968 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:42.014631 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:42.014598 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:42.115154 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:42.115130 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:42.262299 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:42.262257 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:42.366050 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:42.365984 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:42.700575 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:42.700546 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:42.792833 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:42.792811 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:43.258052 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.258017 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:43.258560 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.258540 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2fhzl"] Sep 11 00:58:43.281882 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.281861 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2fhzl"] Sep 11 00:58:43.281987 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.281958 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:43.284460 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.284436 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 00:58:43.284587 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.284490 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 00:58:43.284817 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.284796 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-swjh6\"" Sep 11 00:58:43.285001 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.284982 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 00:58:43.285299 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.285282 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 00:58:43.321685 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.321663 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:43.435389 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.435365 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e048904e-6a28-4df4-8d94-a50b3e15e161-data-volume\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:43.435464 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.435399 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e048904e-6a28-4df4-8d94-a50b3e15e161-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:43.435520 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.435502 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e048904e-6a28-4df4-8d94-a50b3e15e161-crio-socket\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:43.435556 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.435528 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl7zh\" (UniqueName: \"kubernetes.io/projected/e048904e-6a28-4df4-8d94-a50b3e15e161-kube-api-access-rl7zh\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:43.435611 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.435564 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e048904e-6a28-4df4-8d94-a50b3e15e161-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:43.536196 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.536144 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e048904e-6a28-4df4-8d94-a50b3e15e161-data-volume\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:43.536196 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.536182 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e048904e-6a28-4df4-8d94-a50b3e15e161-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:43.536305 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.536259 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e048904e-6a28-4df4-8d94-a50b3e15e161-crio-socket\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:43.536305 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.536276 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl7zh\" (UniqueName: \"kubernetes.io/projected/e048904e-6a28-4df4-8d94-a50b3e15e161-kube-api-access-rl7zh\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:43.536305 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.536294 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e048904e-6a28-4df4-8d94-a50b3e15e161-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:43.536443 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:43.536418 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 00:58:43.536443 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:43.536429 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 00:58:43.536443 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.536431 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e048904e-6a28-4df4-8d94-a50b3e15e161-crio-socket\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:43.536443 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.536438 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e048904e-6a28-4df4-8d94-a50b3e15e161-data-volume\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:43.536633 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:43.536485 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e048904e-6a28-4df4-8d94-a50b3e15e161-insights-runtime-extractor-tls podName:e048904e-6a28-4df4-8d94-a50b3e15e161 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:44.03646797 +0000 UTC m=+50.859921459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e048904e-6a28-4df4-8d94-a50b3e15e161-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2fhzl" (UID: "e048904e-6a28-4df4-8d94-a50b3e15e161") : secret "insights-runtime-extractor-tls" not found Sep 11 00:58:43.536633 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:43.536498 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e048904e-6a28-4df4-8d94-a50b3e15e161-kube-rbac-proxy-cm podName:e048904e-6a28-4df4-8d94-a50b3e15e161 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:44.036491898 +0000 UTC m=+50.859945387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e048904e-6a28-4df4-8d94-a50b3e15e161-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2fhzl" (UID: "e048904e-6a28-4df4-8d94-a50b3e15e161") : configmap "kube-rbac-proxy" not found Sep 11 00:58:43.545393 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.545371 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl7zh\" (UniqueName: \"kubernetes.io/projected/e048904e-6a28-4df4-8d94-a50b3e15e161-kube-api-access-rl7zh\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:43.838837 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.838777 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bd830e3-7d02-4637-b2b7-cf06baad6a26-metrics-tls\") pod \"dns-default-vbxc5\" (UID: \"1bd830e3-7d02-4637-b2b7-cf06baad6a26\") " pod="openshift-dns/dns-default-vbxc5" Sep 11 00:58:43.838937 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:43.838911 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 00:58:43.838982 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:43.838965 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bd830e3-7d02-4637-b2b7-cf06baad6a26-metrics-tls podName:1bd830e3-7d02-4637-b2b7-cf06baad6a26 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:59.838953082 +0000 UTC m=+66.662406569 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1bd830e3-7d02-4637-b2b7-cf06baad6a26-metrics-tls") pod "dns-default-vbxc5" (UID: "1bd830e3-7d02-4637-b2b7-cf06baad6a26") : secret "dns-default-metrics-tls" not found Sep 11 00:58:43.897774 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.897748 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-g9p6x"] Sep 11 00:58:43.899603 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.899586 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-g9p6x" Sep 11 00:58:43.904005 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.903975 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 11 00:58:43.904114 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.903975 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 11 00:58:43.904114 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.904057 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 11 00:58:43.904610 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.904592 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 11 00:58:43.904747 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.904728 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9ksrt\"" Sep 11 00:58:43.908703 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.908667 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-g9p6x"] Sep 11 00:58:43.934123 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.934096 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:43.992974 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:43.992952 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:44.039979 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:44.039956 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e048904e-6a28-4df4-8d94-a50b3e15e161-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:44.040071 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:44.040006 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml8hn\" (UniqueName: \"kubernetes.io/projected/6cd21be1-b455-410c-ade0-5ff04a00ab80-kube-api-access-ml8hn\") pod \"service-ca-568869fdc5-g9p6x\" (UID: \"6cd21be1-b455-410c-ade0-5ff04a00ab80\") " pod="openshift-service-ca/service-ca-568869fdc5-g9p6x" Sep 11 00:58:44.040071 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:44.040061 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 00:58:44.040179 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:44.040091 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e048904e-6a28-4df4-8d94-a50b3e15e161-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:44.040179 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:44.040123 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e048904e-6a28-4df4-8d94-a50b3e15e161-kube-rbac-proxy-cm podName:e048904e-6a28-4df4-8d94-a50b3e15e161 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:45.040106157 +0000 UTC m=+51.863559648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e048904e-6a28-4df4-8d94-a50b3e15e161-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2fhzl" (UID: "e048904e-6a28-4df4-8d94-a50b3e15e161") : configmap "kube-rbac-proxy" not found Sep 11 00:58:44.040289 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:44.040176 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6cd21be1-b455-410c-ade0-5ff04a00ab80-signing-cabundle\") pod \"service-ca-568869fdc5-g9p6x\" (UID: \"6cd21be1-b455-410c-ade0-5ff04a00ab80\") " pod="openshift-service-ca/service-ca-568869fdc5-g9p6x" Sep 11 00:58:44.040289 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:44.040183 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 00:58:44.040289 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:44.040205 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6cd21be1-b455-410c-ade0-5ff04a00ab80-signing-key\") pod \"service-ca-568869fdc5-g9p6x\" (UID: \"6cd21be1-b455-410c-ade0-5ff04a00ab80\") " pod="openshift-service-ca/service-ca-568869fdc5-g9p6x" Sep 11 00:58:44.040289 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:44.040235 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e048904e-6a28-4df4-8d94-a50b3e15e161-insights-runtime-extractor-tls podName:e048904e-6a28-4df4-8d94-a50b3e15e161 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:45.040223559 +0000 UTC m=+51.863677061 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e048904e-6a28-4df4-8d94-a50b3e15e161-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2fhzl" (UID: "e048904e-6a28-4df4-8d94-a50b3e15e161") : secret "insights-runtime-extractor-tls" not found Sep 11 00:58:44.141203 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:44.141156 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml8hn\" (UniqueName: \"kubernetes.io/projected/6cd21be1-b455-410c-ade0-5ff04a00ab80-kube-api-access-ml8hn\") pod \"service-ca-568869fdc5-g9p6x\" (UID: \"6cd21be1-b455-410c-ade0-5ff04a00ab80\") " pod="openshift-service-ca/service-ca-568869fdc5-g9p6x" Sep 11 00:58:44.141281 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:44.141232 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6cd21be1-b455-410c-ade0-5ff04a00ab80-signing-cabundle\") pod \"service-ca-568869fdc5-g9p6x\" (UID: \"6cd21be1-b455-410c-ade0-5ff04a00ab80\") " pod="openshift-service-ca/service-ca-568869fdc5-g9p6x" Sep 11 00:58:44.141281 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:44.141251 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6cd21be1-b455-410c-ade0-5ff04a00ab80-signing-key\") pod \"service-ca-568869fdc5-g9p6x\" (UID: \"6cd21be1-b455-410c-ade0-5ff04a00ab80\") " pod="openshift-service-ca/service-ca-568869fdc5-g9p6x" Sep 11 00:58:44.141833 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:44.141814 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6cd21be1-b455-410c-ade0-5ff04a00ab80-signing-cabundle\") pod \"service-ca-568869fdc5-g9p6x\" (UID: \"6cd21be1-b455-410c-ade0-5ff04a00ab80\") " pod="openshift-service-ca/service-ca-568869fdc5-g9p6x" Sep 11 00:58:44.143764 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:44.143739 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6cd21be1-b455-410c-ade0-5ff04a00ab80-signing-key\") pod \"service-ca-568869fdc5-g9p6x\" (UID: \"6cd21be1-b455-410c-ade0-5ff04a00ab80\") " pod="openshift-service-ca/service-ca-568869fdc5-g9p6x" Sep 11 00:58:44.149709 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:44.149685 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml8hn\" (UniqueName: \"kubernetes.io/projected/6cd21be1-b455-410c-ade0-5ff04a00ab80-kube-api-access-ml8hn\") pod \"service-ca-568869fdc5-g9p6x\" (UID: \"6cd21be1-b455-410c-ade0-5ff04a00ab80\") " pod="openshift-service-ca/service-ca-568869fdc5-g9p6x" Sep 11 00:58:44.212918 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:44.212887 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-g9p6x" Sep 11 00:58:44.481455 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:44.481432 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-g9p6x"] Sep 11 00:58:44.483607 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:58:44.483582 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd21be1_b455_410c_ade0_5ff04a00ab80.slice/crio-cedd9064761cea289f32dd0aa8d5a48a305068846d639a436198ab93be614abe WatchSource:0}: Error finding container cedd9064761cea289f32dd0aa8d5a48a305068846d639a436198ab93be614abe: Status 404 returned error can't find the container with id cedd9064761cea289f32dd0aa8d5a48a305068846d639a436198ab93be614abe Sep 11 00:58:44.840824 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:44.840740 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:44.941341 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:44.941301 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-g9p6x" event={"ID":"6cd21be1-b455-410c-ade0-5ff04a00ab80","Type":"ContainerStarted","Data":"cedd9064761cea289f32dd0aa8d5a48a305068846d639a436198ab93be614abe"} Sep 11 00:58:45.048415 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:45.048385 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e048904e-6a28-4df4-8d94-a50b3e15e161-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:45.048561 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:45.048475 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e048904e-6a28-4df4-8d94-a50b3e15e161-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:45.048628 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:45.048556 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 00:58:45.048628 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:45.048577 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 00:58:45.048724 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:45.048631 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e048904e-6a28-4df4-8d94-a50b3e15e161-insights-runtime-extractor-tls podName:e048904e-6a28-4df4-8d94-a50b3e15e161 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:47.048611005 +0000 UTC m=+53.872064497 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e048904e-6a28-4df4-8d94-a50b3e15e161-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2fhzl" (UID: "e048904e-6a28-4df4-8d94-a50b3e15e161") : secret "insights-runtime-extractor-tls" not found Sep 11 00:58:45.048724 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:45.048653 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e048904e-6a28-4df4-8d94-a50b3e15e161-kube-rbac-proxy-cm podName:e048904e-6a28-4df4-8d94-a50b3e15e161 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:47.048643562 +0000 UTC m=+53.872097055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e048904e-6a28-4df4-8d94-a50b3e15e161-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2fhzl" (UID: "e048904e-6a28-4df4-8d94-a50b3e15e161") : configmap "kube-rbac-proxy" not found Sep 11 00:58:45.148923 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:45.148833 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b619e4ba-cf0c-4078-b918-3dbd3f67c119-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-tqxxh\" (UID: \"b619e4ba-cf0c-4078-b918-3dbd3f67c119\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tqxxh" Sep 11 00:58:45.149055 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:45.148996 2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 00:58:45.149113 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:45.149067 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b619e4ba-cf0c-4078-b918-3dbd3f67c119-cluster-monitoring-operator-tls podName:b619e4ba-cf0c-4078-b918-3dbd3f67c119 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:53.149049106 +0000 UTC m=+59.972502597 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b619e4ba-cf0c-4078-b918-3dbd3f67c119-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-tqxxh" (UID: "b619e4ba-cf0c-4078-b918-3dbd3f67c119") : secret "cluster-monitoring-operator-tls" not found Sep 11 00:58:45.249561 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:45.249526 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dee19a50-296e-4a89-9e02-21ee8aa3ca87-service-ca-bundle\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:45.249727 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:45.249589 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee19a50-296e-4a89-9e02-21ee8aa3ca87-metrics-certs\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:45.249727 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:45.249687 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dee19a50-296e-4a89-9e02-21ee8aa3ca87-service-ca-bundle podName:dee19a50-296e-4a89-9e02-21ee8aa3ca87 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:53.249671837 +0000 UTC m=+60.073125338 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dee19a50-296e-4a89-9e02-21ee8aa3ca87-service-ca-bundle") pod "router-default-5946c9f774-tk9pg" (UID: "dee19a50-296e-4a89-9e02-21ee8aa3ca87") : configmap references non-existent config key: service-ca.crt Sep 11 00:58:45.249870 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:45.249761 2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 00:58:45.249870 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:45.249817 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee19a50-296e-4a89-9e02-21ee8aa3ca87-metrics-certs podName:dee19a50-296e-4a89-9e02-21ee8aa3ca87 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:53.249801016 +0000 UTC m=+60.073254505 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dee19a50-296e-4a89-9e02-21ee8aa3ca87-metrics-certs") pod "router-default-5946c9f774-tk9pg" (UID: "dee19a50-296e-4a89-9e02-21ee8aa3ca87") : secret "router-metrics-certs-default" not found Sep 11 00:58:45.350877 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:45.350832 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/746635c6-8538-4d95-ad45-7f573caee4b3-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-r7p6n\" (UID: \"746635c6-8538-4d95-ad45-7f573caee4b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r7p6n" Sep 11 00:58:45.351028 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:45.350968 2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 00:58:45.351028 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:45.351021 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/746635c6-8538-4d95-ad45-7f573caee4b3-networking-console-plugin-cert podName:746635c6-8538-4d95-ad45-7f573caee4b3 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:53.351008219 +0000 UTC m=+60.174461721 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/746635c6-8538-4d95-ad45-7f573caee4b3-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-r7p6n" (UID: "746635c6-8538-4d95-ad45-7f573caee4b3") : secret "networking-console-plugin-cert" not found Sep 11 00:58:46.212635 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:46.212609 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qxf8s_35c8be8a-e0b7-4843-b8db-4b6a486aee9e/dns-node-resolver/0.log" Sep 11 00:58:46.948200 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:46.948168 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-g9p6x" event={"ID":"6cd21be1-b455-410c-ade0-5ff04a00ab80","Type":"ContainerStarted","Data":"7b966d3b0f592a336ea33834c47e4c8354c8133b12dcae34206b4600e9b50ebd"} Sep 11 00:58:46.967253 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:46.967204 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-g9p6x" podStartSLOduration=2.061204369 podStartE2EDuration="3.967184803s" totalImagesPullingTime="1.905980434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:58:46.966378295 +0000 UTC m=+53.789831806" watchObservedRunningTime="2026-09-11 00:58:46.967184803 +0000 UTC m=+53.790638315" Sep 11 00:58:47.064312 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:47.064283 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e048904e-6a28-4df4-8d94-a50b3e15e161-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:47.064415 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:47.064382 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 00:58:47.064415 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:47.064396 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e048904e-6a28-4df4-8d94-a50b3e15e161-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:47.064510 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:47.064435 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e048904e-6a28-4df4-8d94-a50b3e15e161-kube-rbac-proxy-cm podName:e048904e-6a28-4df4-8d94-a50b3e15e161 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:51.06442002 +0000 UTC m=+57.887873526 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e048904e-6a28-4df4-8d94-a50b3e15e161-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2fhzl" (UID: "e048904e-6a28-4df4-8d94-a50b3e15e161") : configmap "kube-rbac-proxy" not found Sep 11 00:58:47.064510 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:47.064485 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 00:58:47.064600 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:47.064524 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e048904e-6a28-4df4-8d94-a50b3e15e161-insights-runtime-extractor-tls podName:e048904e-6a28-4df4-8d94-a50b3e15e161 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:51.064512118 +0000 UTC m=+57.887965620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e048904e-6a28-4df4-8d94-a50b3e15e161-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2fhzl" (UID: "e048904e-6a28-4df4-8d94-a50b3e15e161") : secret "insights-runtime-extractor-tls" not found Sep 11 00:58:47.211185 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:47.211126 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-spz6s_70820178-37d7-4c9e-8240-0905cdfe37ee/node-ca/0.log" Sep 11 00:58:48.019113 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:48.019082 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ffbjw_a04fe26a-2690-4d67-bd7a-c19c9d610589/serve-healthcheck-canary/0.log" Sep 11 00:58:48.111099 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:48.111066 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:48.202194 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:48.202164 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:48.339935 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:48.339862 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:48.410716 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:48.410692 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:48.499099 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:48.499075 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:48.599554 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:48.599480 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:48.613746 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:48.613718 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-mng4p_2145b039-e727-4271-ad8e-e15b459b2a24/kube-storage-version-migrator-operator/0.log" Sep 11 00:58:48.873503 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:48.873432 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:49.979498 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:49.979466 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:50.178859 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:50.178815 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:50.382785 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:50.382694 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:50.575517 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:50.575482 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:50.771108 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:50.771082 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:51.097921 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:51.097854 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e048904e-6a28-4df4-8d94-a50b3e15e161-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:51.098290 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:51.097958 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 00:58:51.098290 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:51.098016 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e048904e-6a28-4df4-8d94-a50b3e15e161-kube-rbac-proxy-cm podName:e048904e-6a28-4df4-8d94-a50b3e15e161 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:59.09799927 +0000 UTC m=+65.921452774 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e048904e-6a28-4df4-8d94-a50b3e15e161-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2fhzl" (UID: "e048904e-6a28-4df4-8d94-a50b3e15e161") : configmap "kube-rbac-proxy" not found Sep 11 00:58:51.098290 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:51.097958 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e048904e-6a28-4df4-8d94-a50b3e15e161-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:51.098290 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:51.098033 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 00:58:51.098290 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:51.098164 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e048904e-6a28-4df4-8d94-a50b3e15e161-insights-runtime-extractor-tls podName:e048904e-6a28-4df4-8d94-a50b3e15e161 nodeName:}" failed. No retries permitted until 2026-09-11 00:58:59.098150886 +0000 UTC m=+65.921604392 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e048904e-6a28-4df4-8d94-a50b3e15e161-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2fhzl" (UID: "e048904e-6a28-4df4-8d94-a50b3e15e161") : secret "insights-runtime-extractor-tls" not found Sep 11 00:58:51.770011 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:51.769974 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:51.833358 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:51.833327 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:52.177194 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:52.177163 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:52.369951 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:52.369911 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:52.579890 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:52.579799 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:52.641060 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:52.641038 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:52.874950 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:52.874896 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wtd8v" Sep 11 00:58:52.970078 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:52.970058 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:53.215944 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:53.215921 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b619e4ba-cf0c-4078-b918-3dbd3f67c119-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-tqxxh\" (UID: \"b619e4ba-cf0c-4078-b918-3dbd3f67c119\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tqxxh" Sep 11 00:58:53.216367 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:53.216058 2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 00:58:53.216367 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:53.216117 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b619e4ba-cf0c-4078-b918-3dbd3f67c119-cluster-monitoring-operator-tls podName:b619e4ba-cf0c-4078-b918-3dbd3f67c119 nodeName:}" failed. No retries permitted until 2026-09-11 00:59:09.216099696 +0000 UTC m=+76.039553204 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b619e4ba-cf0c-4078-b918-3dbd3f67c119-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-tqxxh" (UID: "b619e4ba-cf0c-4078-b918-3dbd3f67c119") : secret "cluster-monitoring-operator-tls" not found Sep 11 00:58:53.316490 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:53.316462 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dee19a50-296e-4a89-9e02-21ee8aa3ca87-service-ca-bundle\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:53.316587 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:53.316513 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee19a50-296e-4a89-9e02-21ee8aa3ca87-metrics-certs\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:53.316634 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:53.316600 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dee19a50-296e-4a89-9e02-21ee8aa3ca87-service-ca-bundle podName:dee19a50-296e-4a89-9e02-21ee8aa3ca87 nodeName:}" failed. No retries permitted until 2026-09-11 00:59:09.316585834 +0000 UTC m=+76.140039337 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dee19a50-296e-4a89-9e02-21ee8aa3ca87-service-ca-bundle") pod "router-default-5946c9f774-tk9pg" (UID: "dee19a50-296e-4a89-9e02-21ee8aa3ca87") : configmap references non-existent config key: service-ca.crt Sep 11 00:58:53.318736 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:53.318713 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dee19a50-296e-4a89-9e02-21ee8aa3ca87-metrics-certs\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:58:53.417274 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:53.417254 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/746635c6-8538-4d95-ad45-7f573caee4b3-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-r7p6n\" (UID: \"746635c6-8538-4d95-ad45-7f573caee4b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r7p6n" Sep 11 00:58:53.417384 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:53.417369 2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 00:58:53.417425 ip-10-0-135-107 kubenswrapper[2554]: E0911 00:58:53.417414 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/746635c6-8538-4d95-ad45-7f573caee4b3-networking-console-plugin-cert podName:746635c6-8538-4d95-ad45-7f573caee4b3 nodeName:}" failed. No retries permitted until 2026-09-11 00:59:09.417403067 +0000 UTC m=+76.240856577 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/746635c6-8538-4d95-ad45-7f573caee4b3-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-r7p6n" (UID: "746635c6-8538-4d95-ad45-7f573caee4b3") : secret "networking-console-plugin-cert" not found Sep 11 00:58:53.573797 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:53.573741 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:53.971372 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:53.971341 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:54.172292 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:54.172260 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:54.272382 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:54.272318 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:54.571336 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:54.571270 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:54.878452 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:54.878373 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:55.172900 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:55.172869 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:55.773373 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:55.773341 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:55.975310 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:55.975276 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:56.174687 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:56.174658 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:56.481108 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:56.481049 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:56.581758 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:56.581704 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:57.170634 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:57.170606 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:57.572689 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:57.572618 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:57.978857 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:57.978812 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:58.388622 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:58.388553 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:58.453004 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:58.452977 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-metrics-certs\") pod \"network-metrics-daemon-87kmh\" (UID: \"8f0c9d36-ae86-4625-8dd1-0c9764c655e5\") " pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:58:58.453108 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:58.453087 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-original-pull-secret\") pod \"global-pull-secret-syncer-9rbhz\" (UID: \"6038fbb3-56b0-49d0-875a-8f21cfdc9e77\") " pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:58:58.455515 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:58.455491 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 00:58:58.455577 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:58.455535 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 00:58:58.465696 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:58.465671 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f0c9d36-ae86-4625-8dd1-0c9764c655e5-metrics-certs\") pod \"network-metrics-daemon-87kmh\" (UID: \"8f0c9d36-ae86-4625-8dd1-0c9764c655e5\") " pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:58:58.465790 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:58.465702 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6038fbb3-56b0-49d0-875a-8f21cfdc9e77-original-pull-secret\") pod \"global-pull-secret-syncer-9rbhz\" (UID: \"6038fbb3-56b0-49d0-875a-8f21cfdc9e77\") " pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:58:58.553870 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:58.553831 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78czw\" (UniqueName: \"kubernetes.io/projected/d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e-kube-api-access-78czw\") pod \"network-check-target-dwb7w\" (UID: \"d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e\") " pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:58:58.556799 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:58.556783 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 00:58:58.566924 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:58.566907 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 00:58:58.573281 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:58.573255 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:58.577194 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:58.577170 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78czw\" (UniqueName: \"kubernetes.io/projected/d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e-kube-api-access-78czw\") pod \"network-check-target-dwb7w\" (UID: \"d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e\") " pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:58:58.681455 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:58.681437 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rbhz" Sep 11 00:58:58.689348 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:58.689330 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jwmm5\"" Sep 11 00:58:58.693238 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:58.693218 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lb8gv\"" Sep 11 00:58:58.697199 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:58.697178 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:58:58.702004 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:58.701977 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87kmh" Sep 11 00:58:58.778142 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:58.778082 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:58:58.953340 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:58.953286 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9rbhz"] Sep 11 00:58:58.956270 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:58:58.956246 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6038fbb3_56b0_49d0_875a_8f21cfdc9e77.slice/crio-9b7ed532d9ae06f04ded32444d9ca419b96c6a5f0e54f4fcfa680401a3f655fc WatchSource:0}: Error finding container 9b7ed532d9ae06f04ded32444d9ca419b96c6a5f0e54f4fcfa680401a3f655fc: Status 404 returned error can't find the container with id 9b7ed532d9ae06f04ded32444d9ca419b96c6a5f0e54f4fcfa680401a3f655fc Sep 11 00:58:58.974714 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:58.974687 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dwb7w"] Sep 11 00:58:58.976916 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:58.976883 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9rbhz" event={"ID":"6038fbb3-56b0-49d0-875a-8f21cfdc9e77","Type":"ContainerStarted","Data":"9b7ed532d9ae06f04ded32444d9ca419b96c6a5f0e54f4fcfa680401a3f655fc"} Sep 11 00:58:58.977940 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:58.977913 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-87kmh"] Sep 11 00:58:58.978166 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:58:58.978143 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e36d24_9f84_4bdd_a64f_c5e9ec7e5f6e.slice/crio-ad9478f50cc3faacd554c324835a195a35ac4aa02c7f22c48b2c26aa13b50f2f WatchSource:0}: Error finding container ad9478f50cc3faacd554c324835a195a35ac4aa02c7f22c48b2c26aa13b50f2f: Status 404 returned error can't find the container with id ad9478f50cc3faacd554c324835a195a35ac4aa02c7f22c48b2c26aa13b50f2f Sep 11 00:58:58.980472 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:58:58.980454 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0c9d36_ae86_4625_8dd1_0c9764c655e5.slice/crio-57d2bf441a1f86e1d9de4af2d22ec7b68f10949c624092212a4e8399914a3ff9 WatchSource:0}: Error finding container 57d2bf441a1f86e1d9de4af2d22ec7b68f10949c624092212a4e8399914a3ff9: Status 404 returned error can't find the container with id 57d2bf441a1f86e1d9de4af2d22ec7b68f10949c624092212a4e8399914a3ff9 Sep 11 00:58:59.158455 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:59.158421 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e048904e-6a28-4df4-8d94-a50b3e15e161-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:59.158573 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:59.158493 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e048904e-6a28-4df4-8d94-a50b3e15e161-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:59.158974 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:59.158951 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e048904e-6a28-4df4-8d94-a50b3e15e161-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:59.160621 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:59.160604 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e048904e-6a28-4df4-8d94-a50b3e15e161-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2fhzl\" (UID: \"e048904e-6a28-4df4-8d94-a50b3e15e161\") " pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:59.192988 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:59.192961 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-swjh6\"" Sep 11 00:58:59.201104 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:59.201089 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2fhzl" Sep 11 00:58:59.469338 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:59.469261 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2fhzl"] Sep 11 00:58:59.473029 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:58:59.473001 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode048904e_6a28_4df4_8d94_a50b3e15e161.slice/crio-d96b2ec73152ac8bf42544f8ff04ec261b56813ef399cffe6374583964b82c78 WatchSource:0}: Error finding container d96b2ec73152ac8bf42544f8ff04ec261b56813ef399cffe6374583964b82c78: Status 404 returned error can't find the container with id d96b2ec73152ac8bf42544f8ff04ec261b56813ef399cffe6374583964b82c78 Sep 11 00:58:59.864383 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:59.864290 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bd830e3-7d02-4637-b2b7-cf06baad6a26-metrics-tls\") pod \"dns-default-vbxc5\" (UID: \"1bd830e3-7d02-4637-b2b7-cf06baad6a26\") " pod="openshift-dns/dns-default-vbxc5" Sep 11 00:58:59.869546 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:59.867834 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bd830e3-7d02-4637-b2b7-cf06baad6a26-metrics-tls\") pod \"dns-default-vbxc5\" (UID: \"1bd830e3-7d02-4637-b2b7-cf06baad6a26\") " pod="openshift-dns/dns-default-vbxc5" Sep 11 00:58:59.985046 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:59.984894 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2fhzl" event={"ID":"e048904e-6a28-4df4-8d94-a50b3e15e161","Type":"ContainerStarted","Data":"e3529250b9d49d4b94eb0ffe2b79a42c3fabea97c91a0bf52d23064174f8e5ec"} Sep 11 00:58:59.985046 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:59.985009 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2fhzl" event={"ID":"e048904e-6a28-4df4-8d94-a50b3e15e161","Type":"ContainerStarted","Data":"d96b2ec73152ac8bf42544f8ff04ec261b56813ef399cffe6374583964b82c78"} Sep 11 00:58:59.986502 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:59.986474 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-87kmh" event={"ID":"8f0c9d36-ae86-4625-8dd1-0c9764c655e5","Type":"ContainerStarted","Data":"57d2bf441a1f86e1d9de4af2d22ec7b68f10949c624092212a4e8399914a3ff9"} Sep 11 00:58:59.989237 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:58:59.989199 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dwb7w" event={"ID":"d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e","Type":"ContainerStarted","Data":"ad9478f50cc3faacd554c324835a195a35ac4aa02c7f22c48b2c26aa13b50f2f"} Sep 11 00:59:00.087790 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:00.087759 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q9kdp\"" Sep 11 00:59:00.095724 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:00.095690 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vbxc5" Sep 11 00:59:00.413084 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:00.413040 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vbxc5"] Sep 11 00:59:00.417055 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:59:00.417023 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd830e3_7d02_4637_b2b7_cf06baad6a26.slice/crio-1a6ff5cf0c2a9d79ceb8fd8cfc1d9d9bf344917e7cfb8af06ffb81ee00cc8213 WatchSource:0}: Error finding container 1a6ff5cf0c2a9d79ceb8fd8cfc1d9d9bf344917e7cfb8af06ffb81ee00cc8213: Status 404 returned error can't find the container with id 1a6ff5cf0c2a9d79ceb8fd8cfc1d9d9bf344917e7cfb8af06ffb81ee00cc8213 Sep 11 00:59:00.995801 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:00.995763 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2fhzl" event={"ID":"e048904e-6a28-4df4-8d94-a50b3e15e161","Type":"ContainerStarted","Data":"f325d6226004df6e5e1a97c72de00415fcc41a938c2dcea44529999cfd799bd8"} Sep 11 00:59:00.999489 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:00.998681 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-87kmh" event={"ID":"8f0c9d36-ae86-4625-8dd1-0c9764c655e5","Type":"ContainerStarted","Data":"34efa55fe6fa59fa565c08adb110dd840904d6f547d50685e5ce53924a172fa6"} Sep 11 00:59:00.999489 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:00.998734 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-87kmh" event={"ID":"8f0c9d36-ae86-4625-8dd1-0c9764c655e5","Type":"ContainerStarted","Data":"641ddebefe0939d9669498cdeb09ced7ed992a77d51f909b467137b1fc05b3e8"} Sep 11 00:59:01.002447 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:01.002385 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vbxc5" event={"ID":"1bd830e3-7d02-4637-b2b7-cf06baad6a26","Type":"ContainerStarted","Data":"1a6ff5cf0c2a9d79ceb8fd8cfc1d9d9bf344917e7cfb8af06ffb81ee00cc8213"} Sep 11 00:59:01.016532 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:01.015602 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-87kmh" podStartSLOduration=66.832844015 podStartE2EDuration="1m8.015582566s" totalImagesPullingTime="1.182738551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:01.01489748 +0000 UTC m=+67.838350991" watchObservedRunningTime="2026-09-11 00:59:01.015582566 +0000 UTC m=+67.839036080" Sep 11 00:59:01.411708 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:01.411630 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:01.514338 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:01.514299 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:01.613664 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:01.613633 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:01.910303 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:01.910245 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:02.092761 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:02.092723 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:02.153975 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:02.153942 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:02.311627 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:02.311594 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:02.375432 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:02.375395 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:02.717035 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:02.717001 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:02.775519 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:02.775484 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:03.097018 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:03.096944 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:03.294641 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:03.294592 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:03.493085 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:03.493052 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:04.014783 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:04.014745 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:04.513199 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:04.513168 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:04.564647 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:04.564613 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:05.020231 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:05.020141 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vbxc5" event={"ID":"1bd830e3-7d02-4637-b2b7-cf06baad6a26","Type":"ContainerStarted","Data":"05dcb35e392c12e3a1693654cce447a3f34abd4a815642483bde2c2b9cb181ad"} Sep 11 00:59:05.020366 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:05.020248 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vbxc5" Sep 11 00:59:05.020366 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:05.020276 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vbxc5" event={"ID":"1bd830e3-7d02-4637-b2b7-cf06baad6a26","Type":"ContainerStarted","Data":"5889113b1d30ef1cb3e168e95fcf26eca82323a478fd6c863c5c8b118d352c0f"} Sep 11 00:59:05.022299 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:05.022274 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2fhzl" event={"ID":"e048904e-6a28-4df4-8d94-a50b3e15e161","Type":"ContainerStarted","Data":"a6c7412fd0d6f7b1c0156112d502a47dd61e980bb3461083d52c3b7eaa67778c"} Sep 11 00:59:05.023750 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:05.023721 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9rbhz" event={"ID":"6038fbb3-56b0-49d0-875a-8f21cfdc9e77","Type":"ContainerStarted","Data":"21f4f766cc6f3bf0c44cac7f62a4d9c6a9eb01b1bb106eeda2804c5302da2cec"} Sep 11 00:59:05.025309 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:05.025287 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dwb7w" event={"ID":"d5e36d24-9f84-4bdd-a64f-c5e9ec7e5f6e","Type":"ContainerStarted","Data":"fa26ea266d35a24901d5c8037dfd013624c11fa46455ee8f0bf0c6dea7088cba"} Sep 11 00:59:05.025420 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:05.025408 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:59:05.043493 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:05.043451 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vbxc5" podStartSLOduration=33.726411241 podStartE2EDuration="38.04343869s" totalImagesPullingTime="4.317027449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:05.042453851 +0000 UTC m=+71.865907360" watchObservedRunningTime="2026-09-11 00:59:05.04343869 +0000 UTC m=+71.866892199" Sep 11 00:59:05.060898 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:05.060838 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9rbhz" podStartSLOduration=66.24405321 podStartE2EDuration="1m12.0608263s" totalImagesPullingTime="5.81677309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:05.060044646 +0000 UTC m=+71.883498155" watchObservedRunningTime="2026-09-11 00:59:05.0608263 +0000 UTC m=+71.884279809" Sep 11 00:59:05.119604 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:05.119569 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2fhzl" podStartSLOduration=16.976764099 podStartE2EDuration="22.119556597s" totalImagesPullingTime="5.142792498s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:58:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:05.094999137 +0000 UTC m=+71.918452647" watchObservedRunningTime="2026-09-11 00:59:05.119556597 +0000 UTC m=+71.943010107" Sep 11 00:59:05.119736 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:05.119719 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dwb7w" podStartSLOduration=66.338541671 podStartE2EDuration="1m12.119715033s" totalImagesPullingTime="5.781173362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:05.119339588 +0000 UTC m=+71.942793099" watchObservedRunningTime="2026-09-11 00:59:05.119715033 +0000 UTC m=+71.943168543" Sep 11 00:59:06.513556 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.513521 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:06.588910 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.588878 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:06.762127 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.762102 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd"] Sep 11 00:59:06.766060 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.766007 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" Sep 11 00:59:06.768853 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.768820 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 11 00:59:06.769194 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.769178 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 11 00:59:06.769801 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.769778 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 11 00:59:06.769918 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.769824 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 11 00:59:06.770296 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.770278 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 11 00:59:06.770403 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.770297 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 11 00:59:06.770403 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.770308 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 11 00:59:06.788021 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.788001 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd"] Sep 11 00:59:06.820632 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.820604 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ccd0bf-c53d-47f4-ae75-19f787614a68-additional-service-ca\") pod \"cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd\" (UID: \"72ccd0bf-c53d-47f4-ae75-19f787614a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" Sep 11 00:59:06.820727 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.820650 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/72ccd0bf-c53d-47f4-ae75-19f787614a68-hub\") pod \"cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd\" (UID: \"72ccd0bf-c53d-47f4-ae75-19f787614a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" Sep 11 00:59:06.820727 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.820677 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hflbt\" (UniqueName: \"kubernetes.io/projected/72ccd0bf-c53d-47f4-ae75-19f787614a68-kube-api-access-hflbt\") pod \"cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd\" (UID: \"72ccd0bf-c53d-47f4-ae75-19f787614a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" Sep 11 00:59:06.820816 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.820762 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/72ccd0bf-c53d-47f4-ae75-19f787614a68-ca\") pod \"cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd\" (UID: \"72ccd0bf-c53d-47f4-ae75-19f787614a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" Sep 11 00:59:06.820875 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.820819 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/72ccd0bf-c53d-47f4-ae75-19f787614a68-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd\" (UID: \"72ccd0bf-c53d-47f4-ae75-19f787614a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" Sep 11 00:59:06.820918 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.820883 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/72ccd0bf-c53d-47f4-ae75-19f787614a68-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd\" (UID: \"72ccd0bf-c53d-47f4-ae75-19f787614a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" Sep 11 00:59:06.921984 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.921955 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/72ccd0bf-c53d-47f4-ae75-19f787614a68-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd\" (UID: \"72ccd0bf-c53d-47f4-ae75-19f787614a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" Sep 11 00:59:06.922092 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.922012 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/72ccd0bf-c53d-47f4-ae75-19f787614a68-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd\" (UID: \"72ccd0bf-c53d-47f4-ae75-19f787614a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" Sep 11 00:59:06.922092 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.922049 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ccd0bf-c53d-47f4-ae75-19f787614a68-additional-service-ca\") pod \"cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd\" (UID: \"72ccd0bf-c53d-47f4-ae75-19f787614a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" Sep 11 00:59:06.922215 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.922096 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/72ccd0bf-c53d-47f4-ae75-19f787614a68-hub\") pod \"cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd\" (UID: \"72ccd0bf-c53d-47f4-ae75-19f787614a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" Sep 11 00:59:06.922215 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.922134 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hflbt\" (UniqueName: \"kubernetes.io/projected/72ccd0bf-c53d-47f4-ae75-19f787614a68-kube-api-access-hflbt\") pod \"cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd\" (UID: \"72ccd0bf-c53d-47f4-ae75-19f787614a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" Sep 11 00:59:06.922215 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.922163 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/72ccd0bf-c53d-47f4-ae75-19f787614a68-ca\") pod \"cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd\" (UID: \"72ccd0bf-c53d-47f4-ae75-19f787614a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" Sep 11 00:59:06.922805 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.922776 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ccd0bf-c53d-47f4-ae75-19f787614a68-additional-service-ca\") pod \"cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd\" (UID: \"72ccd0bf-c53d-47f4-ae75-19f787614a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" Sep 11 00:59:06.924500 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.924471 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/72ccd0bf-c53d-47f4-ae75-19f787614a68-ca\") pod \"cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd\" (UID: \"72ccd0bf-c53d-47f4-ae75-19f787614a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" Sep 11 00:59:06.924648 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.924627 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/72ccd0bf-c53d-47f4-ae75-19f787614a68-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd\" (UID: \"72ccd0bf-c53d-47f4-ae75-19f787614a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" Sep 11 00:59:06.924713 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.924652 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/72ccd0bf-c53d-47f4-ae75-19f787614a68-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd\" (UID: \"72ccd0bf-c53d-47f4-ae75-19f787614a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" Sep 11 00:59:06.924777 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.924752 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/72ccd0bf-c53d-47f4-ae75-19f787614a68-hub\") pod \"cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd\" (UID: \"72ccd0bf-c53d-47f4-ae75-19f787614a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" Sep 11 00:59:06.937170 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.937148 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:06.938005 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:06.937982 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hflbt\" (UniqueName: \"kubernetes.io/projected/72ccd0bf-c53d-47f4-ae75-19f787614a68-kube-api-access-hflbt\") pod \"cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd\" (UID: \"72ccd0bf-c53d-47f4-ae75-19f787614a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" Sep 11 00:59:07.014220 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:07.014191 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:07.092491 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:07.092430 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" Sep 11 00:59:07.114911 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:07.114882 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:07.217705 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:07.217675 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:07.363709 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:07.363640 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd"] Sep 11 00:59:07.367400 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:59:07.367366 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72ccd0bf_c53d_47f4_ae75_19f787614a68.slice/crio-e448363817041519a428e91ba2c6c9fee5665967df9d31212d1f42cecee7a067 WatchSource:0}: Error finding container e448363817041519a428e91ba2c6c9fee5665967df9d31212d1f42cecee7a067: Status 404 returned error can't find the container with id e448363817041519a428e91ba2c6c9fee5665967df9d31212d1f42cecee7a067 Sep 11 00:59:08.038177 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:08.038114 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" event={"ID":"72ccd0bf-c53d-47f4-ae75-19f787614a68","Type":"ContainerStarted","Data":"e448363817041519a428e91ba2c6c9fee5665967df9d31212d1f42cecee7a067"} Sep 11 00:59:09.240764 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:09.240727 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b619e4ba-cf0c-4078-b918-3dbd3f67c119-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-tqxxh\" (UID: \"b619e4ba-cf0c-4078-b918-3dbd3f67c119\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tqxxh" Sep 11 00:59:09.243581 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:09.243549 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b619e4ba-cf0c-4078-b918-3dbd3f67c119-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-tqxxh\" (UID: \"b619e4ba-cf0c-4078-b918-3dbd3f67c119\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tqxxh" Sep 11 00:59:09.341577 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:09.341543 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dee19a50-296e-4a89-9e02-21ee8aa3ca87-service-ca-bundle\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:59:09.342333 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:09.342287 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dee19a50-296e-4a89-9e02-21ee8aa3ca87-service-ca-bundle\") pod \"router-default-5946c9f774-tk9pg\" (UID: \"dee19a50-296e-4a89-9e02-21ee8aa3ca87\") " pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:59:09.442828 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:09.442796 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/746635c6-8538-4d95-ad45-7f573caee4b3-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-r7p6n\" (UID: \"746635c6-8538-4d95-ad45-7f573caee4b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r7p6n" Sep 11 00:59:09.445209 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:09.445184 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/746635c6-8538-4d95-ad45-7f573caee4b3-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-r7p6n\" (UID: \"746635c6-8538-4d95-ad45-7f573caee4b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r7p6n" Sep 11 00:59:09.481499 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:09.481471 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pnh88\"" Sep 11 00:59:09.489654 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:09.489636 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tqxxh" Sep 11 00:59:09.588451 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:09.588428 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-nhkb9\"" Sep 11 00:59:09.596314 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:09.596299 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:59:09.654042 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:09.654013 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-cd5bm\"" Sep 11 00:59:09.661959 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:09.661929 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r7p6n" Sep 11 00:59:09.765629 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:09.765607 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-tqxxh"] Sep 11 00:59:10.675283 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:59:10.675253 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb619e4ba_cf0c_4078_b918_3dbd3f67c119.slice/crio-da302bee03965782291e3c400d4f4347bf9c4b3f30ce3de63a1b4e87836edd02 WatchSource:0}: Error finding container da302bee03965782291e3c400d4f4347bf9c4b3f30ce3de63a1b4e87836edd02: Status 404 returned error can't find the container with id da302bee03965782291e3c400d4f4347bf9c4b3f30ce3de63a1b4e87836edd02 Sep 11 00:59:10.942455 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:10.942383 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-r7p6n"] Sep 11 00:59:10.944684 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:59:10.944656 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746635c6_8538_4d95_ad45_7f573caee4b3.slice/crio-6cc01cc8b8179bbd689b73dde91a78aeee1b154e7d6fbf23a4ca3b5624beb859 WatchSource:0}: Error finding container 6cc01cc8b8179bbd689b73dde91a78aeee1b154e7d6fbf23a4ca3b5624beb859: Status 404 returned error can't find the container with id 6cc01cc8b8179bbd689b73dde91a78aeee1b154e7d6fbf23a4ca3b5624beb859 Sep 11 00:59:10.946904 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:10.946880 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5946c9f774-tk9pg"] Sep 11 00:59:10.949792 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:59:10.949769 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee19a50_296e_4a89_9e02_21ee8aa3ca87.slice/crio-35a0a0fcc78115b626d6b7e4fda36c49d03c3449dcedf994513170adb92d1fa5 WatchSource:0}: Error finding container 35a0a0fcc78115b626d6b7e4fda36c49d03c3449dcedf994513170adb92d1fa5: Status 404 returned error can't find the container with id 35a0a0fcc78115b626d6b7e4fda36c49d03c3449dcedf994513170adb92d1fa5 Sep 11 00:59:11.050375 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:11.050345 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tqxxh" event={"ID":"b619e4ba-cf0c-4078-b918-3dbd3f67c119","Type":"ContainerStarted","Data":"da302bee03965782291e3c400d4f4347bf9c4b3f30ce3de63a1b4e87836edd02"} Sep 11 00:59:11.052729 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:11.052697 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" event={"ID":"72ccd0bf-c53d-47f4-ae75-19f787614a68","Type":"ContainerStarted","Data":"0c68aa8d326be788cc63b1a173fa8218fd49c5786d9842078ca34da5fea18857"} Sep 11 00:59:11.052890 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:11.052742 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" event={"ID":"72ccd0bf-c53d-47f4-ae75-19f787614a68","Type":"ContainerStarted","Data":"0bbe84c0608cb4ac9c85451f215616da036b1d6449b896851b90c0e9d3fb2b86"} Sep 11 00:59:11.052890 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:11.052760 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" event={"ID":"72ccd0bf-c53d-47f4-ae75-19f787614a68","Type":"ContainerStarted","Data":"9a8565af084fd9b6d900a7bde2b922b9c22a14a27ad220945b7f567b6af94244"} Sep 11 00:59:11.054194 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:11.054171 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5946c9f774-tk9pg" event={"ID":"dee19a50-296e-4a89-9e02-21ee8aa3ca87","Type":"ContainerStarted","Data":"d387dc4fc20eef0082ed2d43b7ca9c40a520c4dfc37ba807dd3f6ccf1a1d8b0e"} Sep 11 00:59:11.054300 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:11.054207 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5946c9f774-tk9pg" event={"ID":"dee19a50-296e-4a89-9e02-21ee8aa3ca87","Type":"ContainerStarted","Data":"35a0a0fcc78115b626d6b7e4fda36c49d03c3449dcedf994513170adb92d1fa5"} Sep 11 00:59:11.055314 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:11.055295 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r7p6n" event={"ID":"746635c6-8538-4d95-ad45-7f573caee4b3","Type":"ContainerStarted","Data":"6cc01cc8b8179bbd689b73dde91a78aeee1b154e7d6fbf23a4ca3b5624beb859"} Sep 11 00:59:11.072110 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:11.072066 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68f6bd8ff4-sn5nd" podStartSLOduration=1.69458218 podStartE2EDuration="5.072051271s" totalImagesPullingTime="3.377469091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:11.071418683 +0000 UTC m=+77.894872193" watchObservedRunningTime="2026-09-11 00:59:11.072051271 +0000 UTC m=+77.895504777" Sep 11 00:59:11.090857 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:11.090802 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5946c9f774-tk9pg" podStartSLOduration=34.090785276 podStartE2EDuration="34.090785276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:11.090043266 +0000 UTC m=+77.913496776" watchObservedRunningTime="2026-09-11 00:59:11.090785276 +0000 UTC m=+77.914238785" Sep 11 00:59:11.597527 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:11.597489 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:59:11.600551 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:11.600525 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:59:12.028947 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:12.028909 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:12.059255 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:12.059223 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:59:12.060355 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:12.060333 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5946c9f774-tk9pg" Sep 11 00:59:12.130086 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:12.130054 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:12.762448 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:12.762371 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:12.862591 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:12.862567 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:12.935553 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:12.935513 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:13.064112 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:13.064049 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r7p6n" event={"ID":"746635c6-8538-4d95-ad45-7f573caee4b3","Type":"ContainerStarted","Data":"0b9f176815df8ad8b2c629fb5cb12edfc6dafcf3d5b730035c0f763da86ebb7f"} Sep 11 00:59:13.065786 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:13.065763 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tqxxh" event={"ID":"b619e4ba-cf0c-4078-b918-3dbd3f67c119","Type":"ContainerStarted","Data":"68c3337528d68b55ec6e0674a1af1014a1000956e083837516f2c786c9aa4a54"} Sep 11 00:59:13.079634 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:13.079582 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-r7p6n" podStartSLOduration=34.586354852 podStartE2EDuration="36.079568954s" totalImagesPullingTime="1.493214102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:13.078902148 +0000 UTC m=+79.902355664" watchObservedRunningTime="2026-09-11 00:59:13.079568954 +0000 UTC m=+79.903022464" Sep 11 00:59:13.096624 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:13.096583 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-tqxxh" podStartSLOduration=34.331758788 podStartE2EDuration="36.096569566s" totalImagesPullingTime="1.764810778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:13.095415535 +0000 UTC m=+79.918869046" watchObservedRunningTime="2026-09-11 00:59:13.096569566 +0000 UTC m=+79.920023075" Sep 11 00:59:13.184383 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:13.184255 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:13.284671 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:13.284633 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:13.724925 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:13.724889 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:13.823333 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:13.823309 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:15.030908 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:15.030873 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vbxc5" Sep 11 00:59:15.149279 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:15.149239 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:15.210196 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:15.210155 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:15.758523 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:15.758487 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:15.811543 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:15.811515 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:16.948571 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:16.948538 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:17.033144 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:17.033110 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:17.130741 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:17.130706 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:18.946241 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:18.946211 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:20.095661 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:20.095629 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:20.168437 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:20.168395 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:20.314750 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:20.314717 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:20.366558 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:20.366472 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:21.096950 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.096912 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:21.323017 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.322957 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:21.359571 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.359505 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gwbnp"] Sep 11 00:59:21.362272 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.362252 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.366073 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.366049 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 00:59:21.366232 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.366215 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 00:59:21.366375 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.366357 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 00:59:21.366375 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.366372 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t5pv6\"" Sep 11 00:59:21.366548 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.366442 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 00:59:21.425315 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.425289 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6236f436-04e9-49e1-8ac1-238b296432a6-node-exporter-wtmp\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.425433 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.425329 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6236f436-04e9-49e1-8ac1-238b296432a6-node-exporter-textfile\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.425433 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.425355 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6236f436-04e9-49e1-8ac1-238b296432a6-sys\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.425522 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.425480 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6236f436-04e9-49e1-8ac1-238b296432a6-metrics-client-ca\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.425573 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.425521 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6236f436-04e9-49e1-8ac1-238b296432a6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.425630 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.425593 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6236f436-04e9-49e1-8ac1-238b296432a6-node-exporter-accelerators-collector-config\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.425685 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.425631 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6236f436-04e9-49e1-8ac1-238b296432a6-root\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.425685 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.425653 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj4vw\" (UniqueName: \"kubernetes.io/projected/6236f436-04e9-49e1-8ac1-238b296432a6-kube-api-access-vj4vw\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.425770 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.425691 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6236f436-04e9-49e1-8ac1-238b296432a6-node-exporter-tls\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.429155 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.429079 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:21.514562 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.514534 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:21.526545 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.526519 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6236f436-04e9-49e1-8ac1-238b296432a6-root\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.526633 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.526548 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj4vw\" (UniqueName: \"kubernetes.io/projected/6236f436-04e9-49e1-8ac1-238b296432a6-kube-api-access-vj4vw\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.526633 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.526568 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6236f436-04e9-49e1-8ac1-238b296432a6-node-exporter-tls\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.526633 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.526586 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6236f436-04e9-49e1-8ac1-238b296432a6-node-exporter-wtmp\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.526633 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.526612 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6236f436-04e9-49e1-8ac1-238b296432a6-node-exporter-textfile\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.526838 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.526635 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6236f436-04e9-49e1-8ac1-238b296432a6-sys\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.526838 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.526660 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6236f436-04e9-49e1-8ac1-238b296432a6-root\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.526838 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.526680 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6236f436-04e9-49e1-8ac1-238b296432a6-metrics-client-ca\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.526838 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.526708 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6236f436-04e9-49e1-8ac1-238b296432a6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.526838 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.526745 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6236f436-04e9-49e1-8ac1-238b296432a6-node-exporter-accelerators-collector-config\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.526838 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.526787 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6236f436-04e9-49e1-8ac1-238b296432a6-node-exporter-wtmp\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.526838 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.526835 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6236f436-04e9-49e1-8ac1-238b296432a6-sys\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.527198 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.527152 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6236f436-04e9-49e1-8ac1-238b296432a6-node-exporter-textfile\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.527412 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.527391 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6236f436-04e9-49e1-8ac1-238b296432a6-node-exporter-accelerators-collector-config\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.527479 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.527404 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6236f436-04e9-49e1-8ac1-238b296432a6-metrics-client-ca\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.529184 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.529163 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6236f436-04e9-49e1-8ac1-238b296432a6-node-exporter-tls\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.529265 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.529252 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6236f436-04e9-49e1-8ac1-238b296432a6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.536913 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.536892 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj4vw\" (UniqueName: \"kubernetes.io/projected/6236f436-04e9-49e1-8ac1-238b296432a6-kube-api-access-vj4vw\") pod \"node-exporter-gwbnp\" (UID: \"6236f436-04e9-49e1-8ac1-238b296432a6\") " pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.620253 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.620187 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:21.672421 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.672394 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gwbnp" Sep 11 00:59:21.717165 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.717138 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:21.815914 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:21.815883 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:22.018746 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:22.018720 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:22.098604 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:22.098578 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gwbnp" event={"ID":"6236f436-04e9-49e1-8ac1-238b296432a6","Type":"ContainerStarted","Data":"2edb9f12bf4c23f49d2264e50d6d6cbf9d13e724d0b7f88ba1757352aaa98595"} Sep 11 00:59:22.285131 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:22.285054 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:22.377331 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:22.377289 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:22.467904 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:22.467868 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:22.529537 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:22.529502 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:22.720960 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:22.720829 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:22.821568 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:22.821541 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:23.104456 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:23.104430 2554 generic.go:356] "Generic (PLEG): container finished" podID="6236f436-04e9-49e1-8ac1-238b296432a6" containerID="be55fc64b4971f0e17245b7adb2ac0f9e563064257502d61527ee96dc6eb024f" exitCode=0 Sep 11 00:59:23.104802 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:23.104500 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gwbnp" event={"ID":"6236f436-04e9-49e1-8ac1-238b296432a6","Type":"ContainerDied","Data":"be55fc64b4971f0e17245b7adb2ac0f9e563064257502d61527ee96dc6eb024f"} Sep 11 00:59:24.112020 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:24.111991 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gwbnp" event={"ID":"6236f436-04e9-49e1-8ac1-238b296432a6","Type":"ContainerStarted","Data":"e9f18ac6d36eaea2e993a4686e759f1eae8496d57587e0cbb216279e94438786"} Sep 11 00:59:24.112419 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:24.112039 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gwbnp" event={"ID":"6236f436-04e9-49e1-8ac1-238b296432a6","Type":"ContainerStarted","Data":"5987a55446c24c6c5acd6688f4630d094e3be7a55b633ce79e83e71f06136542"} Sep 11 00:59:24.130830 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:24.130774 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gwbnp" podStartSLOduration=2.137458531 podStartE2EDuration="3.130754868s" totalImagesPullingTime="930.831049ms" totalInitContainerRuntime="62.465288ms" isStatefulPod=true podCreationTimestamp="2026-09-11 00:59:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:24.130557271 +0000 UTC m=+90.954010781" watchObservedRunningTime="2026-09-11 00:59:24.130754868 +0000 UTC m=+90.954208378" Sep 11 00:59:24.294480 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:24.294449 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:24.374836 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:24.374766 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:24.445509 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:24.445477 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:24.547127 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:24.547104 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:24.913449 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:24.913419 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:25.013990 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:25.013960 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:25.736622 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:25.736585 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:25.843577 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:25.843543 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:26.127877 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:26.127771 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:26.230041 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:26.230010 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:26.387689 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:26.387607 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:27.524169 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:27.524132 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:27.617622 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:27.617588 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:27.739225 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:27.739191 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:29.068751 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:29.068715 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:29.181116 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:29.181084 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:29.277575 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:29.277542 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:31.583792 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.583756 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58c5fc44c6-bpkft"] Sep 11 00:59:31.586263 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.586245 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:31.589353 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.589330 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 00:59:31.589641 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.589619 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 00:59:31.589730 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.589623 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 11 00:59:31.590696 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.590669 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 11 00:59:31.591211 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.591187 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 00:59:31.591211 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.591210 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 00:59:31.591376 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.591240 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 00:59:31.593554 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.592109 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6f9nw\"" Sep 11 00:59:31.599228 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.599206 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58c5fc44c6-bpkft"] Sep 11 00:59:31.599942 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.599921 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 11 00:59:31.698100 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.698075 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ceaba93c-19e4-4e45-aaa7-62d7686af74d-console-config\") pod \"console-58c5fc44c6-bpkft\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:31.698218 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.698106 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ceaba93c-19e4-4e45-aaa7-62d7686af74d-console-oauth-config\") pod \"console-58c5fc44c6-bpkft\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:31.698218 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.698126 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ceaba93c-19e4-4e45-aaa7-62d7686af74d-oauth-serving-cert\") pod \"console-58c5fc44c6-bpkft\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:31.698218 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.698195 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceaba93c-19e4-4e45-aaa7-62d7686af74d-trusted-ca-bundle\") pod \"console-58c5fc44c6-bpkft\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:31.698327 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.698220 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ceaba93c-19e4-4e45-aaa7-62d7686af74d-console-serving-cert\") pod \"console-58c5fc44c6-bpkft\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:31.698327 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.698242 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ceaba93c-19e4-4e45-aaa7-62d7686af74d-service-ca\") pod \"console-58c5fc44c6-bpkft\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:31.698327 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.698300 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v22h\" (UniqueName: \"kubernetes.io/projected/ceaba93c-19e4-4e45-aaa7-62d7686af74d-kube-api-access-7v22h\") pod \"console-58c5fc44c6-bpkft\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:31.744187 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.744162 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:31.799206 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.799179 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ceaba93c-19e4-4e45-aaa7-62d7686af74d-service-ca\") pod \"console-58c5fc44c6-bpkft\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:31.799299 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.799219 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v22h\" (UniqueName: \"kubernetes.io/projected/ceaba93c-19e4-4e45-aaa7-62d7686af74d-kube-api-access-7v22h\") pod \"console-58c5fc44c6-bpkft\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:31.799299 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.799237 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ceaba93c-19e4-4e45-aaa7-62d7686af74d-console-config\") pod \"console-58c5fc44c6-bpkft\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:31.799299 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.799255 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ceaba93c-19e4-4e45-aaa7-62d7686af74d-console-oauth-config\") pod \"console-58c5fc44c6-bpkft\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:31.799299 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.799277 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ceaba93c-19e4-4e45-aaa7-62d7686af74d-oauth-serving-cert\") pod \"console-58c5fc44c6-bpkft\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:31.799514 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.799311 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceaba93c-19e4-4e45-aaa7-62d7686af74d-trusted-ca-bundle\") pod \"console-58c5fc44c6-bpkft\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:31.799514 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.799326 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ceaba93c-19e4-4e45-aaa7-62d7686af74d-console-serving-cert\") pod \"console-58c5fc44c6-bpkft\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:31.799972 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.799946 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ceaba93c-19e4-4e45-aaa7-62d7686af74d-service-ca\") pod \"console-58c5fc44c6-bpkft\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:31.800070 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.800047 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ceaba93c-19e4-4e45-aaa7-62d7686af74d-console-config\") pod \"console-58c5fc44c6-bpkft\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:31.800171 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.800150 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ceaba93c-19e4-4e45-aaa7-62d7686af74d-oauth-serving-cert\") pod \"console-58c5fc44c6-bpkft\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:31.800455 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.800438 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceaba93c-19e4-4e45-aaa7-62d7686af74d-trusted-ca-bundle\") pod \"console-58c5fc44c6-bpkft\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:31.801768 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.801749 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ceaba93c-19e4-4e45-aaa7-62d7686af74d-console-serving-cert\") pod \"console-58c5fc44c6-bpkft\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:31.801768 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.801762 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ceaba93c-19e4-4e45-aaa7-62d7686af74d-console-oauth-config\") pod \"console-58c5fc44c6-bpkft\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:31.806750 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.806728 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v22h\" (UniqueName: \"kubernetes.io/projected/ceaba93c-19e4-4e45-aaa7-62d7686af74d-kube-api-access-7v22h\") pod \"console-58c5fc44c6-bpkft\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:31.847302 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.847253 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:31.899214 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:31.899192 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:32.169448 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:32.169381 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58c5fc44c6-bpkft"] Sep 11 00:59:32.172435 ip-10-0-135-107 kubenswrapper[2554]: W0911 00:59:32.172405 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceaba93c_19e4_4e45_aaa7_62d7686af74d.slice/crio-8f5ab16507a4dc140d6bd271dba9c7739fb8be8ac60fb176b10871d811f51811 WatchSource:0}: Error finding container 8f5ab16507a4dc140d6bd271dba9c7739fb8be8ac60fb176b10871d811f51811: Status 404 returned error can't find the container with id 8f5ab16507a4dc140d6bd271dba9c7739fb8be8ac60fb176b10871d811f51811 Sep 11 00:59:32.354965 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:32.354933 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:32.418538 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:32.418512 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:33.145040 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:33.145004 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58c5fc44c6-bpkft" event={"ID":"ceaba93c-19e4-4e45-aaa7-62d7686af74d","Type":"ContainerStarted","Data":"8f5ab16507a4dc140d6bd271dba9c7739fb8be8ac60fb176b10871d811f51811"} Sep 11 00:59:35.154204 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:35.154126 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58c5fc44c6-bpkft" event={"ID":"ceaba93c-19e4-4e45-aaa7-62d7686af74d","Type":"ContainerStarted","Data":"b9d4be3713554f32a152c7c11dd22fd98f5ebd78a668b9730d6967c4cceded54"} Sep 11 00:59:35.179579 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:35.179532 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58c5fc44c6-bpkft" podStartSLOduration=1.632615591 podStartE2EDuration="4.179517432s" totalImagesPullingTime="2.546901841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:59:35.179273031 +0000 UTC m=+102.002726542" watchObservedRunningTime="2026-09-11 00:59:35.179517432 +0000 UTC m=+102.002970943" Sep 11 00:59:36.031576 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:36.031546 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dwb7w" Sep 11 00:59:36.509261 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:36.509224 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:36.609910 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:36.609875 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:41.900250 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:41.900206 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:41.900627 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:41.900278 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:41.905206 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:41.905181 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:42.115547 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:42.115503 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:42.180202 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:42.180158 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:42.185091 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:42.185070 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 00:59:42.343596 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:42.343558 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:42.460819 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:42.460751 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:51.510830 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:51.510778 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:51.569243 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:51.569211 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:57.678138 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:57.678102 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 00:59:57.737116 ip-10-0-135-107 kubenswrapper[2554]: I0911 00:59:57.737086 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:00:01.247603 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:00:01.247571 2554 generic.go:356] "Generic (PLEG): container finished" podID="2145b039-e727-4271-ad8e-e15b459b2a24" containerID="5afec9c443623a559ce6b652565870066f6d618b8c255e9d1e9e9280cec5f106" exitCode=0 Sep 11 01:00:01.248043 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:00:01.247609 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mng4p" event={"ID":"2145b039-e727-4271-ad8e-e15b459b2a24","Type":"ContainerDied","Data":"5afec9c443623a559ce6b652565870066f6d618b8c255e9d1e9e9280cec5f106"} Sep 11 01:00:01.248043 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:00:01.247906 2554 scope.go:117] "RemoveContainer" containerID="5afec9c443623a559ce6b652565870066f6d618b8c255e9d1e9e9280cec5f106" Sep 11 01:00:02.252626 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:00:02.252594 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-mng4p" event={"ID":"2145b039-e727-4271-ad8e-e15b459b2a24","Type":"ContainerStarted","Data":"e6b137e109dedd73894d6e0506e22431073e63da677564d7f666975e72c7d49e"} Sep 11 01:00:06.513622 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:00:06.513579 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:00:06.832613 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:00:06.832524 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:00:13.943721 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:00:13.943686 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:00:14.053012 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:00:14.052980 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:00:14.145893 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:00:14.145864 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:00:37.448763 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:00:37.448728 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:00:37.504727 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:00:37.504678 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:00:47.780871 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:00:47.780738 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:00:47.835020 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:00:47.834989 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:00:48.806769 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:00:48.806740 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:00:48.862656 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:00:48.862629 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:00:49.403819 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:00:49.403788 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:00:49.464152 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:00:49.464127 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:00:58.436334 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:00:58.436302 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58c5fc44c6-bpkft"] Sep 11 01:00:58.537887 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:00:58.537859 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:00:58.638115 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:00:58.638084 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:00:59.593433 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:00:59.593395 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:01:06.516722 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:06.516661 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:01:06.617581 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:06.617543 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:01:13.808449 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:13.808413 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:01:23.456898 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:23.456826 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-58c5fc44c6-bpkft" podUID="ceaba93c-19e4-4e45-aaa7-62d7686af74d" containerName="console" containerID="cri-o://b9d4be3713554f32a152c7c11dd22fd98f5ebd78a668b9730d6967c4cceded54" gracePeriod=15 Sep 11 01:01:23.689561 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:23.689540 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 01:01:23.749772 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:23.749704 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v22h\" (UniqueName: \"kubernetes.io/projected/ceaba93c-19e4-4e45-aaa7-62d7686af74d-kube-api-access-7v22h\") pod \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " Sep 11 01:01:23.749772 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:23.749757 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ceaba93c-19e4-4e45-aaa7-62d7686af74d-console-oauth-config\") pod \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " Sep 11 01:01:23.749991 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:23.749801 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ceaba93c-19e4-4e45-aaa7-62d7686af74d-oauth-serving-cert\") pod \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " Sep 11 01:01:23.749991 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:23.749823 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceaba93c-19e4-4e45-aaa7-62d7686af74d-trusted-ca-bundle\") pod \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " Sep 11 01:01:23.749991 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:23.749903 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ceaba93c-19e4-4e45-aaa7-62d7686af74d-console-serving-cert\") pod \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " Sep 11 01:01:23.749991 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:23.749939 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ceaba93c-19e4-4e45-aaa7-62d7686af74d-service-ca\") pod \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " Sep 11 01:01:23.749991 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:23.749974 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ceaba93c-19e4-4e45-aaa7-62d7686af74d-console-config\") pod \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\" (UID: \"ceaba93c-19e4-4e45-aaa7-62d7686af74d\") " Sep 11 01:01:23.750294 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:23.750221 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceaba93c-19e4-4e45-aaa7-62d7686af74d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ceaba93c-19e4-4e45-aaa7-62d7686af74d" (UID: "ceaba93c-19e4-4e45-aaa7-62d7686af74d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:01:23.750422 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:23.750351 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceaba93c-19e4-4e45-aaa7-62d7686af74d-service-ca" (OuterVolumeSpecName: "service-ca") pod "ceaba93c-19e4-4e45-aaa7-62d7686af74d" (UID: "ceaba93c-19e4-4e45-aaa7-62d7686af74d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:01:23.750422 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:23.750381 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceaba93c-19e4-4e45-aaa7-62d7686af74d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ceaba93c-19e4-4e45-aaa7-62d7686af74d" (UID: "ceaba93c-19e4-4e45-aaa7-62d7686af74d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:01:23.750541 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:23.750464 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceaba93c-19e4-4e45-aaa7-62d7686af74d-console-config" (OuterVolumeSpecName: "console-config") pod "ceaba93c-19e4-4e45-aaa7-62d7686af74d" (UID: "ceaba93c-19e4-4e45-aaa7-62d7686af74d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:01:23.751969 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:23.751940 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceaba93c-19e4-4e45-aaa7-62d7686af74d-kube-api-access-7v22h" (OuterVolumeSpecName: "kube-api-access-7v22h") pod "ceaba93c-19e4-4e45-aaa7-62d7686af74d" (UID: "ceaba93c-19e4-4e45-aaa7-62d7686af74d"). InnerVolumeSpecName "kube-api-access-7v22h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:01:23.752049 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:23.751997 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceaba93c-19e4-4e45-aaa7-62d7686af74d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ceaba93c-19e4-4e45-aaa7-62d7686af74d" (UID: "ceaba93c-19e4-4e45-aaa7-62d7686af74d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:01:23.752049 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:23.752040 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceaba93c-19e4-4e45-aaa7-62d7686af74d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ceaba93c-19e4-4e45-aaa7-62d7686af74d" (UID: "ceaba93c-19e4-4e45-aaa7-62d7686af74d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:01:23.851153 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:23.851126 2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ceaba93c-19e4-4e45-aaa7-62d7686af74d-console-config\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:01:23.851153 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:23.851154 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v22h\" (UniqueName: \"kubernetes.io/projected/ceaba93c-19e4-4e45-aaa7-62d7686af74d-kube-api-access-7v22h\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:01:23.851269 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:23.851164 2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ceaba93c-19e4-4e45-aaa7-62d7686af74d-console-oauth-config\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:01:23.851269 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:23.851173 2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ceaba93c-19e4-4e45-aaa7-62d7686af74d-oauth-serving-cert\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:01:23.851269 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:23.851181 2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceaba93c-19e4-4e45-aaa7-62d7686af74d-trusted-ca-bundle\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:01:23.851269 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:23.851189 2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ceaba93c-19e4-4e45-aaa7-62d7686af74d-console-serving-cert\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:01:23.851269 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:23.851197 2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ceaba93c-19e4-4e45-aaa7-62d7686af74d-service-ca\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:01:24.536986 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:24.536954 2554 generic.go:356] "Generic (PLEG): container finished" podID="ceaba93c-19e4-4e45-aaa7-62d7686af74d" containerID="b9d4be3713554f32a152c7c11dd22fd98f5ebd78a668b9730d6967c4cceded54" exitCode=0 Sep 11 01:01:24.537367 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:24.536999 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58c5fc44c6-bpkft" Sep 11 01:01:24.537367 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:24.537035 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58c5fc44c6-bpkft" event={"ID":"ceaba93c-19e4-4e45-aaa7-62d7686af74d","Type":"ContainerDied","Data":"b9d4be3713554f32a152c7c11dd22fd98f5ebd78a668b9730d6967c4cceded54"} Sep 11 01:01:24.537367 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:24.537083 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58c5fc44c6-bpkft" event={"ID":"ceaba93c-19e4-4e45-aaa7-62d7686af74d","Type":"ContainerDied","Data":"8f5ab16507a4dc140d6bd271dba9c7739fb8be8ac60fb176b10871d811f51811"} Sep 11 01:01:24.537367 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:24.537103 2554 scope.go:117] "RemoveContainer" containerID="b9d4be3713554f32a152c7c11dd22fd98f5ebd78a668b9730d6967c4cceded54" Sep 11 01:01:24.546046 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:24.546031 2554 scope.go:117] "RemoveContainer" containerID="b9d4be3713554f32a152c7c11dd22fd98f5ebd78a668b9730d6967c4cceded54" Sep 11 01:01:24.546305 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:01:24.546284 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9d4be3713554f32a152c7c11dd22fd98f5ebd78a668b9730d6967c4cceded54\": container with ID starting with b9d4be3713554f32a152c7c11dd22fd98f5ebd78a668b9730d6967c4cceded54 not found: ID does not exist" containerID="b9d4be3713554f32a152c7c11dd22fd98f5ebd78a668b9730d6967c4cceded54" Sep 11 01:01:24.546373 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:24.546316 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9d4be3713554f32a152c7c11dd22fd98f5ebd78a668b9730d6967c4cceded54"} err="failed to get container status \"b9d4be3713554f32a152c7c11dd22fd98f5ebd78a668b9730d6967c4cceded54\": rpc error: code = NotFound desc = could not find container \"b9d4be3713554f32a152c7c11dd22fd98f5ebd78a668b9730d6967c4cceded54\": container with ID starting with b9d4be3713554f32a152c7c11dd22fd98f5ebd78a668b9730d6967c4cceded54 not found: ID does not exist" Sep 11 01:01:24.556329 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:24.556306 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58c5fc44c6-bpkft"] Sep 11 01:01:24.558524 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:24.558506 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58c5fc44c6-bpkft"] Sep 11 01:01:25.141985 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:25.141954 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:01:25.675953 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:25.675923 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceaba93c-19e4-4e45-aaa7-62d7686af74d" path="/var/lib/kubelet/pods/ceaba93c-19e4-4e45-aaa7-62d7686af74d/volumes" Sep 11 01:01:44.845548 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:44.845511 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:01:44.902402 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:44.902369 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:01:56.018235 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:56.018195 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:01:56.111473 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:01:56.111434 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:02:22.907268 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:22.907227 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:02:24.669985 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:24.669947 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:02:26.513411 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:26.513358 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:02:26.610097 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:26.610059 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:02:27.984991 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:27.984954 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:02:28.086425 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:28.086371 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:02:28.280740 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:28.280678 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:02:28.388179 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:28.388150 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:02:30.184987 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:30.184954 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:02:30.575283 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:30.575204 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:02:33.499110 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:33.499061 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:02:33.629246 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:33.629205 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:02:36.615588 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:36.615549 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:02:37.044126 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:37.044092 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:02:37.102188 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:37.102154 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:02:38.574292 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:38.574254 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:02:39.234047 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:39.234014 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7pxdh"] Sep 11 01:02:39.234279 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:39.234267 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceaba93c-19e4-4e45-aaa7-62d7686af74d" containerName="console" Sep 11 01:02:39.234327 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:39.234281 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceaba93c-19e4-4e45-aaa7-62d7686af74d" containerName="console" Sep 11 01:02:39.234362 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:39.234329 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceaba93c-19e4-4e45-aaa7-62d7686af74d" containerName="console" Sep 11 01:02:39.235994 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:39.235978 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-7pxdh" Sep 11 01:02:39.238736 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:39.238714 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 11 01:02:39.240278 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:39.240254 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 11 01:02:39.240418 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:39.240312 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-2htbv\"" Sep 11 01:02:39.247000 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:39.246978 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7pxdh"] Sep 11 01:02:39.342508 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:39.342483 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bcc6bed8-1a8e-4aed-98a5-10e8bc15891e-bound-sa-token\") pod \"cert-manager-55b4d954d5-7pxdh\" (UID: \"bcc6bed8-1a8e-4aed-98a5-10e8bc15891e\") " pod="cert-manager/cert-manager-55b4d954d5-7pxdh" Sep 11 01:02:39.342618 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:39.342525 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4prm\" (UniqueName: \"kubernetes.io/projected/bcc6bed8-1a8e-4aed-98a5-10e8bc15891e-kube-api-access-g4prm\") pod \"cert-manager-55b4d954d5-7pxdh\" (UID: \"bcc6bed8-1a8e-4aed-98a5-10e8bc15891e\") " pod="cert-manager/cert-manager-55b4d954d5-7pxdh" Sep 11 01:02:39.443076 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:39.443053 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4prm\" (UniqueName: \"kubernetes.io/projected/bcc6bed8-1a8e-4aed-98a5-10e8bc15891e-kube-api-access-g4prm\") pod \"cert-manager-55b4d954d5-7pxdh\" (UID: \"bcc6bed8-1a8e-4aed-98a5-10e8bc15891e\") " pod="cert-manager/cert-manager-55b4d954d5-7pxdh" Sep 11 01:02:39.443226 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:39.443205 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bcc6bed8-1a8e-4aed-98a5-10e8bc15891e-bound-sa-token\") pod \"cert-manager-55b4d954d5-7pxdh\" (UID: \"bcc6bed8-1a8e-4aed-98a5-10e8bc15891e\") " pod="cert-manager/cert-manager-55b4d954d5-7pxdh" Sep 11 01:02:39.451580 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:39.451557 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bcc6bed8-1a8e-4aed-98a5-10e8bc15891e-bound-sa-token\") pod \"cert-manager-55b4d954d5-7pxdh\" (UID: \"bcc6bed8-1a8e-4aed-98a5-10e8bc15891e\") " pod="cert-manager/cert-manager-55b4d954d5-7pxdh" Sep 11 01:02:39.451665 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:39.451624 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4prm\" (UniqueName: \"kubernetes.io/projected/bcc6bed8-1a8e-4aed-98a5-10e8bc15891e-kube-api-access-g4prm\") pod \"cert-manager-55b4d954d5-7pxdh\" (UID: \"bcc6bed8-1a8e-4aed-98a5-10e8bc15891e\") " pod="cert-manager/cert-manager-55b4d954d5-7pxdh" Sep 11 01:02:39.545004 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:39.544945 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-7pxdh" Sep 11 01:02:39.809921 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:39.809852 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7pxdh"] Sep 11 01:02:39.812505 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:02:39.812469 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbcc6bed8_1a8e_4aed_98a5_10e8bc15891e.slice/crio-c20afbc25ae702c615bf4730272723aad2acd855afc1c86ad0737030804f300c WatchSource:0}: Error finding container c20afbc25ae702c615bf4730272723aad2acd855afc1c86ad0737030804f300c: Status 404 returned error can't find the container with id c20afbc25ae702c615bf4730272723aad2acd855afc1c86ad0737030804f300c Sep 11 01:02:40.803511 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:40.803450 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-7pxdh" event={"ID":"bcc6bed8-1a8e-4aed-98a5-10e8bc15891e","Type":"ContainerStarted","Data":"c20afbc25ae702c615bf4730272723aad2acd855afc1c86ad0737030804f300c"} Sep 11 01:02:41.917570 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:41.917525 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:02:42.018338 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:42.018302 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:02:43.816205 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:43.816172 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-7pxdh" event={"ID":"bcc6bed8-1a8e-4aed-98a5-10e8bc15891e","Type":"ContainerStarted","Data":"fb8ccf9007a7ba0ce05b2902fd93f3c5c7cbd5f7e050b273618ff3bea3073d31"} Sep 11 01:02:43.831939 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:43.831897 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-7pxdh" podStartSLOduration=1.8533185030000001 podStartE2EDuration="4.831879599s" totalImagesPullingTime="2.978561096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:02:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:02:43.830766552 +0000 UTC m=+290.654220061" watchObservedRunningTime="2026-09-11 01:02:43.831879599 +0000 UTC m=+290.655333110" Sep 11 01:02:47.371209 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:47.371170 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:02:47.469478 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:47.469449 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:02:49.102922 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:49.102885 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mn7v6"] Sep 11 01:02:49.105024 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:49.105004 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mn7v6" Sep 11 01:02:49.107317 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:49.107282 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-mw74c\"" Sep 11 01:02:49.107317 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:49.107311 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 11 01:02:49.108426 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:49.108405 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 11 01:02:49.114176 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:49.114147 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mn7v6"] Sep 11 01:02:49.216639 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:49.216611 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c17083fd-fc2d-4fe4-a34d-963714a3e42c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mn7v6\" (UID: \"c17083fd-fc2d-4fe4-a34d-963714a3e42c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mn7v6" Sep 11 01:02:49.216753 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:49.216647 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfn45\" (UniqueName: \"kubernetes.io/projected/c17083fd-fc2d-4fe4-a34d-963714a3e42c-kube-api-access-sfn45\") pod \"openshift-lws-operator-fd8ccff4c-mn7v6\" (UID: \"c17083fd-fc2d-4fe4-a34d-963714a3e42c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mn7v6" Sep 11 01:02:49.317881 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:49.317834 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c17083fd-fc2d-4fe4-a34d-963714a3e42c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mn7v6\" (UID: \"c17083fd-fc2d-4fe4-a34d-963714a3e42c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mn7v6" Sep 11 01:02:49.317995 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:49.317893 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfn45\" (UniqueName: \"kubernetes.io/projected/c17083fd-fc2d-4fe4-a34d-963714a3e42c-kube-api-access-sfn45\") pod \"openshift-lws-operator-fd8ccff4c-mn7v6\" (UID: \"c17083fd-fc2d-4fe4-a34d-963714a3e42c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mn7v6" Sep 11 01:02:49.318271 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:49.318253 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c17083fd-fc2d-4fe4-a34d-963714a3e42c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mn7v6\" (UID: \"c17083fd-fc2d-4fe4-a34d-963714a3e42c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mn7v6" Sep 11 01:02:49.327483 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:49.327464 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfn45\" (UniqueName: \"kubernetes.io/projected/c17083fd-fc2d-4fe4-a34d-963714a3e42c-kube-api-access-sfn45\") pod \"openshift-lws-operator-fd8ccff4c-mn7v6\" (UID: \"c17083fd-fc2d-4fe4-a34d-963714a3e42c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mn7v6" Sep 11 01:02:49.415572 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:49.415514 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mn7v6" Sep 11 01:02:49.687665 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:49.687638 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mn7v6"] Sep 11 01:02:49.691077 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:02:49.691054 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc17083fd_fc2d_4fe4_a34d_963714a3e42c.slice/crio-6e64fc82d6317a31785c18bc5983d02523bf70167a5e356b84895c432e6c2710 WatchSource:0}: Error finding container 6e64fc82d6317a31785c18bc5983d02523bf70167a5e356b84895c432e6c2710: Status 404 returned error can't find the container with id 6e64fc82d6317a31785c18bc5983d02523bf70167a5e356b84895c432e6c2710 Sep 11 01:02:49.815664 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:49.815638 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:02:49.840236 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:49.840212 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mn7v6" event={"ID":"c17083fd-fc2d-4fe4-a34d-963714a3e42c","Type":"ContainerStarted","Data":"6e64fc82d6317a31785c18bc5983d02523bf70167a5e356b84895c432e6c2710"} Sep 11 01:02:52.853883 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:52.853833 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mn7v6" event={"ID":"c17083fd-fc2d-4fe4-a34d-963714a3e42c","Type":"ContainerStarted","Data":"f467b5b5749e2b178c04b26603f4b37e6b3e4690e07ac83aba2f62281698a7ae"} Sep 11 01:02:52.869515 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:52.869468 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mn7v6" podStartSLOduration=1.2829504219999999 podStartE2EDuration="3.869454309s" totalImagesPullingTime="2.586503887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:02:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:02:52.868363254 +0000 UTC m=+299.691816778" watchObservedRunningTime="2026-09-11 01:02:52.869454309 +0000 UTC m=+299.692907818" Sep 11 01:02:53.556643 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:53.556601 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:02:53.575278 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:53.575250 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wtd8v_c2b320dd-43e9-435d-8e4f-2bca1f1233ff/ovn-acl-logging/0.log" Sep 11 01:02:53.578051 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:53.578027 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wtd8v_c2b320dd-43e9-435d-8e4f-2bca1f1233ff/ovn-acl-logging/0.log" Sep 11 01:02:53.579314 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:53.579293 2554 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 01:02:53.657219 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:53.657099 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:02:55.446709 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:02:55.446673 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:08.138855 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:08.138808 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:09.080235 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.080201 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx"] Sep 11 01:03:09.083582 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.083563 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx" Sep 11 01:03:09.085876 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.085834 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 11 01:03:09.086164 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.086145 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 11 01:03:09.086263 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.086206 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 11 01:03:09.086263 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.086245 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sqsm4\"" Sep 11 01:03:09.086517 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.086502 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 11 01:03:09.105386 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.105365 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx"] Sep 11 01:03:09.155177 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.155150 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5115bd54-60ec-4cd8-9e8e-06436e61adf0-manifests\") pod \"opendatahub-operator-controller-manager-7c49b45946-qpjtx\" (UID: \"5115bd54-60ec-4cd8-9e8e-06436e61adf0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx" Sep 11 01:03:09.155451 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.155179 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml8l4\" (UniqueName: \"kubernetes.io/projected/5115bd54-60ec-4cd8-9e8e-06436e61adf0-kube-api-access-ml8l4\") pod \"opendatahub-operator-controller-manager-7c49b45946-qpjtx\" (UID: \"5115bd54-60ec-4cd8-9e8e-06436e61adf0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx" Sep 11 01:03:09.155451 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.155213 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5115bd54-60ec-4cd8-9e8e-06436e61adf0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c49b45946-qpjtx\" (UID: \"5115bd54-60ec-4cd8-9e8e-06436e61adf0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx" Sep 11 01:03:09.155451 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.155231 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5115bd54-60ec-4cd8-9e8e-06436e61adf0-tmp\") pod \"opendatahub-operator-controller-manager-7c49b45946-qpjtx\" (UID: \"5115bd54-60ec-4cd8-9e8e-06436e61adf0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx" Sep 11 01:03:09.155451 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.155290 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5115bd54-60ec-4cd8-9e8e-06436e61adf0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c49b45946-qpjtx\" (UID: \"5115bd54-60ec-4cd8-9e8e-06436e61adf0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx" Sep 11 01:03:09.219109 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.219079 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:09.255873 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.255834 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5115bd54-60ec-4cd8-9e8e-06436e61adf0-manifests\") pod \"opendatahub-operator-controller-manager-7c49b45946-qpjtx\" (UID: \"5115bd54-60ec-4cd8-9e8e-06436e61adf0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx" Sep 11 01:03:09.255962 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.255878 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml8l4\" (UniqueName: \"kubernetes.io/projected/5115bd54-60ec-4cd8-9e8e-06436e61adf0-kube-api-access-ml8l4\") pod \"opendatahub-operator-controller-manager-7c49b45946-qpjtx\" (UID: \"5115bd54-60ec-4cd8-9e8e-06436e61adf0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx" Sep 11 01:03:09.255962 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.255919 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5115bd54-60ec-4cd8-9e8e-06436e61adf0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c49b45946-qpjtx\" (UID: \"5115bd54-60ec-4cd8-9e8e-06436e61adf0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx" Sep 11 01:03:09.255962 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.255935 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5115bd54-60ec-4cd8-9e8e-06436e61adf0-tmp\") pod \"opendatahub-operator-controller-manager-7c49b45946-qpjtx\" (UID: \"5115bd54-60ec-4cd8-9e8e-06436e61adf0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx" Sep 11 01:03:09.255962 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.255959 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5115bd54-60ec-4cd8-9e8e-06436e61adf0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c49b45946-qpjtx\" (UID: \"5115bd54-60ec-4cd8-9e8e-06436e61adf0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx" Sep 11 01:03:09.256306 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.256271 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5115bd54-60ec-4cd8-9e8e-06436e61adf0-manifests\") pod \"opendatahub-operator-controller-manager-7c49b45946-qpjtx\" (UID: \"5115bd54-60ec-4cd8-9e8e-06436e61adf0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx" Sep 11 01:03:09.258339 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.258308 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5115bd54-60ec-4cd8-9e8e-06436e61adf0-tmp\") pod \"opendatahub-operator-controller-manager-7c49b45946-qpjtx\" (UID: \"5115bd54-60ec-4cd8-9e8e-06436e61adf0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx" Sep 11 01:03:09.258433 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.258402 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5115bd54-60ec-4cd8-9e8e-06436e61adf0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c49b45946-qpjtx\" (UID: \"5115bd54-60ec-4cd8-9e8e-06436e61adf0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx" Sep 11 01:03:09.258485 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.258448 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5115bd54-60ec-4cd8-9e8e-06436e61adf0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c49b45946-qpjtx\" (UID: \"5115bd54-60ec-4cd8-9e8e-06436e61adf0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx" Sep 11 01:03:09.264582 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.264559 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml8l4\" (UniqueName: \"kubernetes.io/projected/5115bd54-60ec-4cd8-9e8e-06436e61adf0-kube-api-access-ml8l4\") pod \"opendatahub-operator-controller-manager-7c49b45946-qpjtx\" (UID: \"5115bd54-60ec-4cd8-9e8e-06436e61adf0\") " pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx" Sep 11 01:03:09.320378 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.320348 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:09.393793 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.393733 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx" Sep 11 01:03:09.421765 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.421705 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:09.522920 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.522889 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:09.620023 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.619994 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:09.674876 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:03:09.674827 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5115bd54_60ec_4cd8_9e8e_06436e61adf0.slice/crio-5c8f2cd78e848bd15a8404b11b917a92c8fc7cfd9c51aee1d9d3a4d82f2961f6 WatchSource:0}: Error finding container 5c8f2cd78e848bd15a8404b11b917a92c8fc7cfd9c51aee1d9d3a4d82f2961f6: Status 404 returned error can't find the container with id 5c8f2cd78e848bd15a8404b11b917a92c8fc7cfd9c51aee1d9d3a4d82f2961f6 Sep 11 01:03:09.676328 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.676307 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx"] Sep 11 01:03:09.676753 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.676739 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 01:03:09.719369 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.719340 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:09.820275 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.820243 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:09.914557 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.914523 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx" event={"ID":"5115bd54-60ec-4cd8-9e8e-06436e61adf0","Type":"ContainerStarted","Data":"5c8f2cd78e848bd15a8404b11b917a92c8fc7cfd9c51aee1d9d3a4d82f2961f6"} Sep 11 01:03:09.921272 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:09.921238 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:10.026772 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:10.026707 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:11.730406 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:11.730368 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:11.827540 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:11.827505 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:12.930610 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:12.930581 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx" event={"ID":"5115bd54-60ec-4cd8-9e8e-06436e61adf0","Type":"ContainerStarted","Data":"aecab8d8ed402f16c7dc82aae125ed92b58cb2b4bced6615108c8ae216522586"} Sep 11 01:03:13.935948 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:13.935915 2554 generic.go:356] "Generic (PLEG): container finished" podID="5115bd54-60ec-4cd8-9e8e-06436e61adf0" containerID="aecab8d8ed402f16c7dc82aae125ed92b58cb2b4bced6615108c8ae216522586" exitCode=0 Sep 11 01:03:13.936351 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:13.936003 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx" event={"ID":"5115bd54-60ec-4cd8-9e8e-06436e61adf0","Type":"ContainerDied","Data":"aecab8d8ed402f16c7dc82aae125ed92b58cb2b4bced6615108c8ae216522586"} Sep 11 01:03:14.943168 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:14.943133 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx" event={"ID":"5115bd54-60ec-4cd8-9e8e-06436e61adf0","Type":"ContainerStarted","Data":"3e367af55dfb5f48ecf49fe1d5b83bc0d525d4827f280b05151580c6ae50a623"} Sep 11 01:03:14.943553 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:14.943262 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx" Sep 11 01:03:14.964644 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:14.964596 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx" podStartSLOduration=1.7413350159999998 podStartE2EDuration="5.964579832s" totalImagesPullingTime="3.219750677s" totalInitContainerRuntime="1.003494139s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:03:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:03:14.963676889 +0000 UTC m=+321.787130398" watchObservedRunningTime="2026-09-11 01:03:14.964579832 +0000 UTC m=+321.788033341" Sep 11 01:03:25.948945 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:25.948916 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7c49b45946-qpjtx" Sep 11 01:03:28.311195 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:28.311162 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:28.546591 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:28.546560 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:29.121058 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:29.121029 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:29.225611 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:29.225571 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5cc5ff74fb-xl5vx"] Sep 11 01:03:29.236472 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:29.236449 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-xl5vx" Sep 11 01:03:29.246005 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:29.245987 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 11 01:03:29.246106 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:29.246088 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5wgf8\"" Sep 11 01:03:29.246772 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:29.246758 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 11 01:03:29.262062 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:29.262040 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cc5ff74fb-xl5vx"] Sep 11 01:03:29.312527 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:29.312497 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:29.396488 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:29.396430 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca046967-4442-4ca5-a5b3-85b5867414f4-tls-certs\") pod \"kube-auth-proxy-5cc5ff74fb-xl5vx\" (UID: \"ca046967-4442-4ca5-a5b3-85b5867414f4\") " pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-xl5vx" Sep 11 01:03:29.396488 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:29.396477 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca046967-4442-4ca5-a5b3-85b5867414f4-tmp\") pod \"kube-auth-proxy-5cc5ff74fb-xl5vx\" (UID: \"ca046967-4442-4ca5-a5b3-85b5867414f4\") " pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-xl5vx" Sep 11 01:03:29.396628 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:29.396499 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgnwc\" (UniqueName: \"kubernetes.io/projected/ca046967-4442-4ca5-a5b3-85b5867414f4-kube-api-access-rgnwc\") pod \"kube-auth-proxy-5cc5ff74fb-xl5vx\" (UID: \"ca046967-4442-4ca5-a5b3-85b5867414f4\") " pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-xl5vx" Sep 11 01:03:29.497121 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:29.497096 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca046967-4442-4ca5-a5b3-85b5867414f4-tls-certs\") pod \"kube-auth-proxy-5cc5ff74fb-xl5vx\" (UID: \"ca046967-4442-4ca5-a5b3-85b5867414f4\") " pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-xl5vx" Sep 11 01:03:29.497220 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:29.497140 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca046967-4442-4ca5-a5b3-85b5867414f4-tmp\") pod \"kube-auth-proxy-5cc5ff74fb-xl5vx\" (UID: \"ca046967-4442-4ca5-a5b3-85b5867414f4\") " pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-xl5vx" Sep 11 01:03:29.497220 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:29.497160 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgnwc\" (UniqueName: \"kubernetes.io/projected/ca046967-4442-4ca5-a5b3-85b5867414f4-kube-api-access-rgnwc\") pod \"kube-auth-proxy-5cc5ff74fb-xl5vx\" (UID: \"ca046967-4442-4ca5-a5b3-85b5867414f4\") " pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-xl5vx" Sep 11 01:03:29.499296 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:29.499277 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca046967-4442-4ca5-a5b3-85b5867414f4-tmp\") pod \"kube-auth-proxy-5cc5ff74fb-xl5vx\" (UID: \"ca046967-4442-4ca5-a5b3-85b5867414f4\") " pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-xl5vx" Sep 11 01:03:29.499523 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:29.499507 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca046967-4442-4ca5-a5b3-85b5867414f4-tls-certs\") pod \"kube-auth-proxy-5cc5ff74fb-xl5vx\" (UID: \"ca046967-4442-4ca5-a5b3-85b5867414f4\") " pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-xl5vx" Sep 11 01:03:29.516660 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:29.516634 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgnwc\" (UniqueName: \"kubernetes.io/projected/ca046967-4442-4ca5-a5b3-85b5867414f4-kube-api-access-rgnwc\") pod \"kube-auth-proxy-5cc5ff74fb-xl5vx\" (UID: \"ca046967-4442-4ca5-a5b3-85b5867414f4\") " pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-xl5vx" Sep 11 01:03:29.545663 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:29.545643 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-xl5vx" Sep 11 01:03:29.837724 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:29.837690 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cc5ff74fb-xl5vx"] Sep 11 01:03:29.842196 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:03:29.842174 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca046967_4442_4ca5_a5b3_85b5867414f4.slice/crio-5fea8a14c92ec9781f214b23c7a02388f170d2bb658adba504dc29dbff933d03 WatchSource:0}: Error finding container 5fea8a14c92ec9781f214b23c7a02388f170d2bb658adba504dc29dbff933d03: Status 404 returned error can't find the container with id 5fea8a14c92ec9781f214b23c7a02388f170d2bb658adba504dc29dbff933d03 Sep 11 01:03:30.002561 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:30.002530 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-xl5vx" event={"ID":"ca046967-4442-4ca5-a5b3-85b5867414f4","Type":"ContainerStarted","Data":"5fea8a14c92ec9781f214b23c7a02388f170d2bb658adba504dc29dbff933d03"} Sep 11 01:03:31.262738 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.262685 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:31.280480 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.280448 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr"] Sep 11 01:03:31.285667 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.285640 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:31.288310 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.288190 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 11 01:03:31.288310 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.288211 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 11 01:03:31.288518 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.288198 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 11 01:03:31.288518 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.288465 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 01:03:31.289401 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.289378 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-twbw8\"" Sep 11 01:03:31.300929 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.300905 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr"] Sep 11 01:03:31.409347 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.409195 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:31.413007 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.412976 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq5sd\" (UniqueName: \"kubernetes.io/projected/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-kube-api-access-qq5sd\") pod \"istiod-openshift-gateway-7b7995d5cd-l57cr\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:31.413141 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.413016 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-l57cr\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:31.413141 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.413044 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-l57cr\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:31.413141 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.413074 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-l57cr\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:31.413141 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.413122 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-l57cr\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:31.413378 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.413167 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-l57cr\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:31.413378 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.413266 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-l57cr\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:31.514469 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.514372 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq5sd\" (UniqueName: \"kubernetes.io/projected/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-kube-api-access-qq5sd\") pod \"istiod-openshift-gateway-7b7995d5cd-l57cr\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:31.514469 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.514417 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-l57cr\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:31.514671 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.514637 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-l57cr\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:31.514728 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.514685 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-l57cr\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:31.514784 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.514729 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-l57cr\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:31.514838 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.514798 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-l57cr\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:31.514929 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.514880 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-l57cr\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:31.515719 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.515671 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-l57cr\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:31.517743 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.517719 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-l57cr\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:31.517864 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.517748 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-l57cr\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:31.517943 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.517917 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-l57cr\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:31.518186 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.518150 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-l57cr\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:31.524328 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.524304 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq5sd\" (UniqueName: \"kubernetes.io/projected/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-kube-api-access-qq5sd\") pod \"istiod-openshift-gateway-7b7995d5cd-l57cr\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:31.524733 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.524704 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-l57cr\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:31.597941 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:31.597920 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:32.476816 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:32.476776 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:32.846329 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:32.846294 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr"] Sep 11 01:03:32.848588 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:03:32.848561 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9d62be_0256_43d3_b47d_1ab4b9b660b9.slice/crio-856d9c433d9dc68c8aa81bd9cd5c1dc31641017ef4e82702d0827e5da2bd7953 WatchSource:0}: Error finding container 856d9c433d9dc68c8aa81bd9cd5c1dc31641017ef4e82702d0827e5da2bd7953: Status 404 returned error can't find the container with id 856d9c433d9dc68c8aa81bd9cd5c1dc31641017ef4e82702d0827e5da2bd7953 Sep 11 01:03:33.016068 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:33.016032 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-xl5vx" event={"ID":"ca046967-4442-4ca5-a5b3-85b5867414f4","Type":"ContainerStarted","Data":"2b5e171e4c881e85eedc7f4772a53f72b02a545156d2cfc4b1e44bb7ed56d6e9"} Sep 11 01:03:33.017328 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:33.017301 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" event={"ID":"8d9d62be-0256-43d3-b47d-1ab4b9b660b9","Type":"ContainerStarted","Data":"856d9c433d9dc68c8aa81bd9cd5c1dc31641017ef4e82702d0827e5da2bd7953"} Sep 11 01:03:33.035268 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:33.035225 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5cc5ff74fb-xl5vx" podStartSLOduration=1.240945929 podStartE2EDuration="4.035207177s" totalImagesPullingTime="2.794261248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:03:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:03:33.034624453 +0000 UTC m=+339.858077962" watchObservedRunningTime="2026-09-11 01:03:33.035207177 +0000 UTC m=+339.858660688" Sep 11 01:03:35.388688 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:35.388652 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 01:03:35.388963 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:35.388722 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 01:03:36.033192 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:36.033151 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" event={"ID":"8d9d62be-0256-43d3-b47d-1ab4b9b660b9","Type":"ContainerStarted","Data":"eaa37c4edd950ce84a96f40c79a8ba92b0a358c7fd180e991ffccfe22c7e7c42"} Sep 11 01:03:36.033382 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:36.033355 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:36.033467 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:36.033394 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:36.034796 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:36.034772 2554 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-l57cr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 01:03:36.034891 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:36.034825 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" podUID="8d9d62be-0256-43d3-b47d-1ab4b9b660b9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 01:03:36.070506 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:36.070457 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" podStartSLOduration=2.532692456 podStartE2EDuration="5.070443108s" totalImagesPullingTime="2.537750652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:03:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:03:36.0690045 +0000 UTC m=+342.892458006" watchObservedRunningTime="2026-09-11 01:03:36.070443108 +0000 UTC m=+342.893896617" Sep 11 01:03:36.142335 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:36.142291 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:36.840533 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:36.840501 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:36.938139 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:36.938099 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:37.035340 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:37.035280 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:37.038828 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:37.038802 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:39.453406 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:39.453354 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:39.519698 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:39.519663 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:39.618587 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:39.618556 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:39.719252 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:39.719185 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:40.048908 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:40.048807 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6mqjx"] Sep 11 01:03:40.052486 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:40.052466 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-6mqjx" Sep 11 01:03:40.055756 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:40.055733 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 11 01:03:40.074347 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:40.074323 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6mqjx"] Sep 11 01:03:40.091928 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:40.091904 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:40.183441 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:40.183414 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8th5\" (UniqueName: \"kubernetes.io/projected/412594fe-d273-4015-87ae-48e593f0f257-kube-api-access-d8th5\") pod \"model-serving-api-54d89bd5fc-6mqjx\" (UID: \"412594fe-d273-4015-87ae-48e593f0f257\") " pod="opendatahub/model-serving-api-54d89bd5fc-6mqjx" Sep 11 01:03:40.183533 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:40.183464 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/412594fe-d273-4015-87ae-48e593f0f257-tls-certs\") pod \"model-serving-api-54d89bd5fc-6mqjx\" (UID: \"412594fe-d273-4015-87ae-48e593f0f257\") " pod="opendatahub/model-serving-api-54d89bd5fc-6mqjx" Sep 11 01:03:40.284097 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:40.284071 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/412594fe-d273-4015-87ae-48e593f0f257-tls-certs\") pod \"model-serving-api-54d89bd5fc-6mqjx\" (UID: \"412594fe-d273-4015-87ae-48e593f0f257\") " pod="opendatahub/model-serving-api-54d89bd5fc-6mqjx" Sep 11 01:03:40.284216 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:40.284137 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8th5\" (UniqueName: \"kubernetes.io/projected/412594fe-d273-4015-87ae-48e593f0f257-kube-api-access-d8th5\") pod \"model-serving-api-54d89bd5fc-6mqjx\" (UID: \"412594fe-d273-4015-87ae-48e593f0f257\") " pod="opendatahub/model-serving-api-54d89bd5fc-6mqjx" Sep 11 01:03:40.284270 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:03:40.284235 2554 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 11 01:03:40.284323 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:03:40.284313 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412594fe-d273-4015-87ae-48e593f0f257-tls-certs podName:412594fe-d273-4015-87ae-48e593f0f257 nodeName:}" failed. No retries permitted until 2026-09-11 01:03:40.78429116 +0000 UTC m=+347.607744665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/412594fe-d273-4015-87ae-48e593f0f257-tls-certs") pod "model-serving-api-54d89bd5fc-6mqjx" (UID: "412594fe-d273-4015-87ae-48e593f0f257") : secret "model-serving-api-tls" not found Sep 11 01:03:40.294398 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:40.294369 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8th5\" (UniqueName: \"kubernetes.io/projected/412594fe-d273-4015-87ae-48e593f0f257-kube-api-access-d8th5\") pod \"model-serving-api-54d89bd5fc-6mqjx\" (UID: \"412594fe-d273-4015-87ae-48e593f0f257\") " pod="opendatahub/model-serving-api-54d89bd5fc-6mqjx" Sep 11 01:03:40.788355 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:40.788314 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/412594fe-d273-4015-87ae-48e593f0f257-tls-certs\") pod \"model-serving-api-54d89bd5fc-6mqjx\" (UID: \"412594fe-d273-4015-87ae-48e593f0f257\") " pod="opendatahub/model-serving-api-54d89bd5fc-6mqjx" Sep 11 01:03:40.788715 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:03:40.788469 2554 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 11 01:03:40.788715 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:03:40.788530 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/412594fe-d273-4015-87ae-48e593f0f257-tls-certs podName:412594fe-d273-4015-87ae-48e593f0f257 nodeName:}" failed. No retries permitted until 2026-09-11 01:03:41.788512761 +0000 UTC m=+348.611966271 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/412594fe-d273-4015-87ae-48e593f0f257-tls-certs") pod "model-serving-api-54d89bd5fc-6mqjx" (UID: "412594fe-d273-4015-87ae-48e593f0f257") : secret "model-serving-api-tls" not found Sep 11 01:03:40.946553 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:40.946522 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:41.798465 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:41.798431 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/412594fe-d273-4015-87ae-48e593f0f257-tls-certs\") pod \"model-serving-api-54d89bd5fc-6mqjx\" (UID: \"412594fe-d273-4015-87ae-48e593f0f257\") " pod="opendatahub/model-serving-api-54d89bd5fc-6mqjx" Sep 11 01:03:41.800837 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:41.800815 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/412594fe-d273-4015-87ae-48e593f0f257-tls-certs\") pod \"model-serving-api-54d89bd5fc-6mqjx\" (UID: \"412594fe-d273-4015-87ae-48e593f0f257\") " pod="opendatahub/model-serving-api-54d89bd5fc-6mqjx" Sep 11 01:03:41.862263 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:41.862239 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-6mqjx" Sep 11 01:03:42.130866 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:42.130822 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6mqjx"] Sep 11 01:03:42.133146 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:03:42.133112 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412594fe_d273_4015_87ae_48e593f0f257.slice/crio-77d54097db22b18306b966648700afb5bb789c3565b8f5660aa651b9661b8ea2 WatchSource:0}: Error finding container 77d54097db22b18306b966648700afb5bb789c3565b8f5660aa651b9661b8ea2: Status 404 returned error can't find the container with id 77d54097db22b18306b966648700afb5bb789c3565b8f5660aa651b9661b8ea2 Sep 11 01:03:43.065233 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:43.065189 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-6mqjx" event={"ID":"412594fe-d273-4015-87ae-48e593f0f257","Type":"ContainerStarted","Data":"77d54097db22b18306b966648700afb5bb789c3565b8f5660aa651b9661b8ea2"} Sep 11 01:03:45.074318 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:45.074278 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-6mqjx" event={"ID":"412594fe-d273-4015-87ae-48e593f0f257","Type":"ContainerStarted","Data":"e78ee78f379c20c2f04e3f1bd2b3a083edb679056b2cdb33f36b2c5e0dc81567"} Sep 11 01:03:45.074736 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:45.074420 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-6mqjx" Sep 11 01:03:45.090966 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:45.090919 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-6mqjx" podStartSLOduration=3.1472016800000002 podStartE2EDuration="5.090904827s" totalImagesPullingTime="1.943703147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:03:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:03:45.089759455 +0000 UTC m=+351.913212959" watchObservedRunningTime="2026-09-11 01:03:45.090904827 +0000 UTC m=+351.914358337" Sep 11 01:03:50.318725 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.318638 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:50.435515 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.435480 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kfzfx"] Sep 11 01:03:50.438527 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.438509 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kfzfx" Sep 11 01:03:50.442207 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.442184 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-2nzh7\"" Sep 11 01:03:50.442335 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.442185 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 11 01:03:50.459730 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.459706 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kfzfx"] Sep 11 01:03:50.465696 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.465675 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-4ktqt"] Sep 11 01:03:50.468805 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.468785 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4ktqt" Sep 11 01:03:50.469047 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.469021 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wcpz\" (UniqueName: \"kubernetes.io/projected/7ddea17b-8323-4ad1-8171-a6e6e00f3133-kube-api-access-5wcpz\") pod \"kserve-controller-manager-d6f4b65b8-kfzfx\" (UID: \"7ddea17b-8323-4ad1-8171-a6e6e00f3133\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kfzfx" Sep 11 01:03:50.469298 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.469281 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ddea17b-8323-4ad1-8171-a6e6e00f3133-cert\") pod \"kserve-controller-manager-d6f4b65b8-kfzfx\" (UID: \"7ddea17b-8323-4ad1-8171-a6e6e00f3133\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kfzfx" Sep 11 01:03:50.472487 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.472469 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-bjp8w\"" Sep 11 01:03:50.472640 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.472622 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 11 01:03:50.494191 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.494167 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-4ktqt"] Sep 11 01:03:50.510583 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.510562 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:50.569725 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.569669 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdzx8\" (UniqueName: \"kubernetes.io/projected/0cf5f7e0-6bb4-4349-9806-42540500a8d2-kube-api-access-sdzx8\") pod \"llmisvc-controller-manager-777c6bd7d5-4ktqt\" (UID: \"0cf5f7e0-6bb4-4349-9806-42540500a8d2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4ktqt" Sep 11 01:03:50.569725 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.569700 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wcpz\" (UniqueName: \"kubernetes.io/projected/7ddea17b-8323-4ad1-8171-a6e6e00f3133-kube-api-access-5wcpz\") pod \"kserve-controller-manager-d6f4b65b8-kfzfx\" (UID: \"7ddea17b-8323-4ad1-8171-a6e6e00f3133\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kfzfx" Sep 11 01:03:50.569725 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.569718 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cf5f7e0-6bb4-4349-9806-42540500a8d2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4ktqt\" (UID: \"0cf5f7e0-6bb4-4349-9806-42540500a8d2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4ktqt" Sep 11 01:03:50.569917 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.569809 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ddea17b-8323-4ad1-8171-a6e6e00f3133-cert\") pod \"kserve-controller-manager-d6f4b65b8-kfzfx\" (UID: \"7ddea17b-8323-4ad1-8171-a6e6e00f3133\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kfzfx" Sep 11 01:03:50.572354 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.572333 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ddea17b-8323-4ad1-8171-a6e6e00f3133-cert\") pod \"kserve-controller-manager-d6f4b65b8-kfzfx\" (UID: \"7ddea17b-8323-4ad1-8171-a6e6e00f3133\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kfzfx" Sep 11 01:03:50.584219 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.584199 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wcpz\" (UniqueName: \"kubernetes.io/projected/7ddea17b-8323-4ad1-8171-a6e6e00f3133-kube-api-access-5wcpz\") pod \"kserve-controller-manager-d6f4b65b8-kfzfx\" (UID: \"7ddea17b-8323-4ad1-8171-a6e6e00f3133\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kfzfx" Sep 11 01:03:50.647650 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.647624 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:50.671575 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.671540 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdzx8\" (UniqueName: \"kubernetes.io/projected/0cf5f7e0-6bb4-4349-9806-42540500a8d2-kube-api-access-sdzx8\") pod \"llmisvc-controller-manager-777c6bd7d5-4ktqt\" (UID: \"0cf5f7e0-6bb4-4349-9806-42540500a8d2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4ktqt" Sep 11 01:03:50.671676 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.671586 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cf5f7e0-6bb4-4349-9806-42540500a8d2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4ktqt\" (UID: \"0cf5f7e0-6bb4-4349-9806-42540500a8d2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4ktqt" Sep 11 01:03:50.671737 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:03:50.671716 2554 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 11 01:03:50.671793 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:03:50.671774 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cf5f7e0-6bb4-4349-9806-42540500a8d2-cert podName:0cf5f7e0-6bb4-4349-9806-42540500a8d2 nodeName:}" failed. No retries permitted until 2026-09-11 01:03:51.171757285 +0000 UTC m=+357.995210784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0cf5f7e0-6bb4-4349-9806-42540500a8d2-cert") pod "llmisvc-controller-manager-777c6bd7d5-4ktqt" (UID: "0cf5f7e0-6bb4-4349-9806-42540500a8d2") : secret "llmisvc-webhook-server-cert" not found Sep 11 01:03:50.686694 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.686670 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdzx8\" (UniqueName: \"kubernetes.io/projected/0cf5f7e0-6bb4-4349-9806-42540500a8d2-kube-api-access-sdzx8\") pod \"llmisvc-controller-manager-777c6bd7d5-4ktqt\" (UID: \"0cf5f7e0-6bb4-4349-9806-42540500a8d2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4ktqt" Sep 11 01:03:50.750436 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:50.750413 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kfzfx" Sep 11 01:03:51.007553 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.007526 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:51.057458 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.057400 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kfzfx"] Sep 11 01:03:51.059507 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:03:51.059484 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ddea17b_8323_4ad1_8171_a6e6e00f3133.slice/crio-a5cf7fc12fb420cf48331f98abc103ba19c915b6fee76a4fac3c39e1cd450fcb WatchSource:0}: Error finding container a5cf7fc12fb420cf48331f98abc103ba19c915b6fee76a4fac3c39e1cd450fcb: Status 404 returned error can't find the container with id a5cf7fc12fb420cf48331f98abc103ba19c915b6fee76a4fac3c39e1cd450fcb Sep 11 01:03:51.099293 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.099266 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kfzfx" event={"ID":"7ddea17b-8323-4ad1-8171-a6e6e00f3133","Type":"ContainerStarted","Data":"a5cf7fc12fb420cf48331f98abc103ba19c915b6fee76a4fac3c39e1cd450fcb"} Sep 11 01:03:51.176605 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.176579 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cf5f7e0-6bb4-4349-9806-42540500a8d2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4ktqt\" (UID: \"0cf5f7e0-6bb4-4349-9806-42540500a8d2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4ktqt" Sep 11 01:03:51.178822 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.178801 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cf5f7e0-6bb4-4349-9806-42540500a8d2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4ktqt\" (UID: \"0cf5f7e0-6bb4-4349-9806-42540500a8d2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4ktqt" Sep 11 01:03:51.300454 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.300379 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr"] Sep 11 01:03:51.300648 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.300604 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" podUID="8d9d62be-0256-43d3-b47d-1ab4b9b660b9" containerName="discovery" containerID="cri-o://eaa37c4edd950ce84a96f40c79a8ba92b0a358c7fd180e991ffccfe22c7e7c42" gracePeriod=30 Sep 11 01:03:51.381035 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.381008 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4ktqt" Sep 11 01:03:51.530342 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.530318 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:51.579269 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.579093 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-cacerts\") pod \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " Sep 11 01:03:51.579269 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.579145 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-local-certs\") pod \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " Sep 11 01:03:51.579269 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.579198 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-istio-kubeconfig\") pod \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " Sep 11 01:03:51.579269 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.579230 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-istio-csr-ca-configmap\") pod \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " Sep 11 01:03:51.579269 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.579258 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-istio-token\") pod \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " Sep 11 01:03:51.579640 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.579306 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-istio-csr-dns-cert\") pod \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " Sep 11 01:03:51.579640 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.579334 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qq5sd\" (UniqueName: \"kubernetes.io/projected/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-kube-api-access-qq5sd\") pod \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\" (UID: \"8d9d62be-0256-43d3-b47d-1ab4b9b660b9\") " Sep 11 01:03:51.579640 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.579608 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "8d9d62be-0256-43d3-b47d-1ab4b9b660b9" (UID: "8d9d62be-0256-43d3-b47d-1ab4b9b660b9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:03:51.581892 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.581864 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "8d9d62be-0256-43d3-b47d-1ab4b9b660b9" (UID: "8d9d62be-0256-43d3-b47d-1ab4b9b660b9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:03:51.581892 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.581902 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-local-certs" (OuterVolumeSpecName: "local-certs") pod "8d9d62be-0256-43d3-b47d-1ab4b9b660b9" (UID: "8d9d62be-0256-43d3-b47d-1ab4b9b660b9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:03:51.582091 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.582021 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "8d9d62be-0256-43d3-b47d-1ab4b9b660b9" (UID: "8d9d62be-0256-43d3-b47d-1ab4b9b660b9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:03:51.582320 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.582298 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-cacerts" (OuterVolumeSpecName: "cacerts") pod "8d9d62be-0256-43d3-b47d-1ab4b9b660b9" (UID: "8d9d62be-0256-43d3-b47d-1ab4b9b660b9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:03:51.582579 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.582554 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-kube-api-access-qq5sd" (OuterVolumeSpecName: "kube-api-access-qq5sd") pod "8d9d62be-0256-43d3-b47d-1ab4b9b660b9" (UID: "8d9d62be-0256-43d3-b47d-1ab4b9b660b9"). InnerVolumeSpecName "kube-api-access-qq5sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:03:51.582881 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.582830 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-istio-token" (OuterVolumeSpecName: "istio-token") pod "8d9d62be-0256-43d3-b47d-1ab4b9b660b9" (UID: "8d9d62be-0256-43d3-b47d-1ab4b9b660b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:03:51.654521 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.654502 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-4ktqt"] Sep 11 01:03:51.656789 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:03:51.656767 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0cf5f7e0_6bb4_4349_9806_42540500a8d2.slice/crio-c9790018f798c17413136b777d728a06d62f605c12aa15fe4a38cfb5c973e649 WatchSource:0}: Error finding container c9790018f798c17413136b777d728a06d62f605c12aa15fe4a38cfb5c973e649: Status 404 returned error can't find the container with id c9790018f798c17413136b777d728a06d62f605c12aa15fe4a38cfb5c973e649 Sep 11 01:03:51.680080 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.680049 2554 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-istio-csr-dns-cert\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:03:51.680080 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.680071 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq5sd\" (UniqueName: \"kubernetes.io/projected/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-kube-api-access-qq5sd\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:03:51.680080 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.680081 2554 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-cacerts\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:03:51.680278 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.680091 2554 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-local-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:03:51.680278 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.680100 2554 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-istio-kubeconfig\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:03:51.680278 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.680107 2554 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-istio-csr-ca-configmap\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:03:51.680278 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.680115 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9d62be-0256-43d3-b47d-1ab4b9b660b9-istio-token\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:03:51.682985 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.682953 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:51.822436 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.822405 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:51.990331 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:51.990300 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:52.105461 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:52.105428 2554 generic.go:356] "Generic (PLEG): container finished" podID="8d9d62be-0256-43d3-b47d-1ab4b9b660b9" containerID="eaa37c4edd950ce84a96f40c79a8ba92b0a358c7fd180e991ffccfe22c7e7c42" exitCode=0 Sep 11 01:03:52.105604 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:52.105473 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" event={"ID":"8d9d62be-0256-43d3-b47d-1ab4b9b660b9","Type":"ContainerDied","Data":"eaa37c4edd950ce84a96f40c79a8ba92b0a358c7fd180e991ffccfe22c7e7c42"} Sep 11 01:03:52.105604 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:52.105495 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" Sep 11 01:03:52.105604 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:52.105528 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr" event={"ID":"8d9d62be-0256-43d3-b47d-1ab4b9b660b9","Type":"ContainerDied","Data":"856d9c433d9dc68c8aa81bd9cd5c1dc31641017ef4e82702d0827e5da2bd7953"} Sep 11 01:03:52.105604 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:52.105553 2554 scope.go:117] "RemoveContainer" containerID="eaa37c4edd950ce84a96f40c79a8ba92b0a358c7fd180e991ffccfe22c7e7c42" Sep 11 01:03:52.107133 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:52.107114 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4ktqt" event={"ID":"0cf5f7e0-6bb4-4349-9806-42540500a8d2","Type":"ContainerStarted","Data":"c9790018f798c17413136b777d728a06d62f605c12aa15fe4a38cfb5c973e649"} Sep 11 01:03:52.115140 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:52.115106 2554 scope.go:117] "RemoveContainer" containerID="eaa37c4edd950ce84a96f40c79a8ba92b0a358c7fd180e991ffccfe22c7e7c42" Sep 11 01:03:52.115437 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:03:52.115411 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa37c4edd950ce84a96f40c79a8ba92b0a358c7fd180e991ffccfe22c7e7c42\": container with ID starting with eaa37c4edd950ce84a96f40c79a8ba92b0a358c7fd180e991ffccfe22c7e7c42 not found: ID does not exist" containerID="eaa37c4edd950ce84a96f40c79a8ba92b0a358c7fd180e991ffccfe22c7e7c42" Sep 11 01:03:52.115525 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:52.115446 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa37c4edd950ce84a96f40c79a8ba92b0a358c7fd180e991ffccfe22c7e7c42"} err="failed to get container status \"eaa37c4edd950ce84a96f40c79a8ba92b0a358c7fd180e991ffccfe22c7e7c42\": rpc error: code = NotFound desc = could not find container \"eaa37c4edd950ce84a96f40c79a8ba92b0a358c7fd180e991ffccfe22c7e7c42\": container with ID starting with eaa37c4edd950ce84a96f40c79a8ba92b0a358c7fd180e991ffccfe22c7e7c42 not found: ID does not exist" Sep 11 01:03:52.123828 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:52.123803 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr"] Sep 11 01:03:52.129282 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:52.129253 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-l57cr"] Sep 11 01:03:53.678545 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:53.678510 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d9d62be-0256-43d3-b47d-1ab4b9b660b9" path="/var/lib/kubelet/pods/8d9d62be-0256-43d3-b47d-1ab4b9b660b9/volumes" Sep 11 01:03:55.124757 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:55.124722 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kfzfx" event={"ID":"7ddea17b-8323-4ad1-8171-a6e6e00f3133","Type":"ContainerStarted","Data":"3caad328d6f458e5f6dcd639b555228a7df86d5ebfe160f9eee861f06cda4ce8"} Sep 11 01:03:55.125228 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:55.124884 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kfzfx" Sep 11 01:03:55.143873 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:55.143810 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kfzfx" podStartSLOduration=2.586288385 podStartE2EDuration="5.14379597s" totalImagesPullingTime="2.557507585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:03:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:03:55.142217225 +0000 UTC m=+361.965670737" watchObservedRunningTime="2026-09-11 01:03:55.14379597 +0000 UTC m=+361.967249480" Sep 11 01:03:56.082309 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:56.082282 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-6mqjx" Sep 11 01:03:56.103257 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:56.103222 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:56.130272 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:56.130236 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4ktqt" event={"ID":"0cf5f7e0-6bb4-4349-9806-42540500a8d2","Type":"ContainerStarted","Data":"95a55d0aaff312ba1c33416181e232163a14f48edc726f1fc97e1b12ec8b3d1b"} Sep 11 01:03:56.130645 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:56.130445 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4ktqt" Sep 11 01:03:56.151107 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:56.151059 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4ktqt" podStartSLOduration=2.583686009 podStartE2EDuration="6.15104615s" totalImagesPullingTime="3.567360141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:03:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:03:56.150477151 +0000 UTC m=+362.973930662" watchObservedRunningTime="2026-09-11 01:03:56.15104615 +0000 UTC m=+362.974499692" Sep 11 01:03:56.207621 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:56.207583 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:58.564215 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:58.564169 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:58.621863 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:58.621814 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:03:59.396048 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:03:59.396008 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:04:26.136443 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:26.136401 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kfzfx" Sep 11 01:04:27.137966 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:27.137934 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4ktqt" Sep 11 01:04:31.986530 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:31.986489 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7685ddfb9f-24w22"] Sep 11 01:04:31.987021 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:31.986936 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9d62be-0256-43d3-b47d-1ab4b9b660b9" containerName="discovery" Sep 11 01:04:31.987021 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:31.986956 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9d62be-0256-43d3-b47d-1ab4b9b660b9" containerName="discovery" Sep 11 01:04:31.987130 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:31.987030 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d9d62be-0256-43d3-b47d-1ab4b9b660b9" containerName="discovery" Sep 11 01:04:31.995190 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:31.995166 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:31.999444 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:31.999420 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 11 01:04:31.999609 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:31.999423 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 01:04:31.999681 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:31.999433 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 01:04:31.999743 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:31.999491 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6f9nw\"" Sep 11 01:04:31.999808 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:31.999493 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 01:04:32.000138 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.000122 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 11 01:04:32.000229 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.000143 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 01:04:32.000229 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.000125 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 01:04:32.006054 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.006028 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7685ddfb9f-24w22"] Sep 11 01:04:32.006816 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.006793 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 11 01:04:32.054938 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.054903 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:04:32.059818 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.059793 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/637e5d57-19bd-4fa5-b4ef-6c3884a5ff43-trusted-ca-bundle\") pod \"console-7685ddfb9f-24w22\" (UID: \"637e5d57-19bd-4fa5-b4ef-6c3884a5ff43\") " pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:32.059955 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.059865 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqxzv\" (UniqueName: \"kubernetes.io/projected/637e5d57-19bd-4fa5-b4ef-6c3884a5ff43-kube-api-access-gqxzv\") pod \"console-7685ddfb9f-24w22\" (UID: \"637e5d57-19bd-4fa5-b4ef-6c3884a5ff43\") " pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:32.059955 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.059952 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/637e5d57-19bd-4fa5-b4ef-6c3884a5ff43-console-serving-cert\") pod \"console-7685ddfb9f-24w22\" (UID: \"637e5d57-19bd-4fa5-b4ef-6c3884a5ff43\") " pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:32.060072 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.060014 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/637e5d57-19bd-4fa5-b4ef-6c3884a5ff43-console-config\") pod \"console-7685ddfb9f-24w22\" (UID: \"637e5d57-19bd-4fa5-b4ef-6c3884a5ff43\") " pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:32.060072 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.060034 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/637e5d57-19bd-4fa5-b4ef-6c3884a5ff43-oauth-serving-cert\") pod \"console-7685ddfb9f-24w22\" (UID: \"637e5d57-19bd-4fa5-b4ef-6c3884a5ff43\") " pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:32.060218 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.060075 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/637e5d57-19bd-4fa5-b4ef-6c3884a5ff43-console-oauth-config\") pod \"console-7685ddfb9f-24w22\" (UID: \"637e5d57-19bd-4fa5-b4ef-6c3884a5ff43\") " pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:32.060218 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.060114 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/637e5d57-19bd-4fa5-b4ef-6c3884a5ff43-service-ca\") pod \"console-7685ddfb9f-24w22\" (UID: \"637e5d57-19bd-4fa5-b4ef-6c3884a5ff43\") " pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:32.113374 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.113343 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:04:32.161207 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.161176 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/637e5d57-19bd-4fa5-b4ef-6c3884a5ff43-console-serving-cert\") pod \"console-7685ddfb9f-24w22\" (UID: \"637e5d57-19bd-4fa5-b4ef-6c3884a5ff43\") " pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:32.161371 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.161214 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/637e5d57-19bd-4fa5-b4ef-6c3884a5ff43-console-config\") pod \"console-7685ddfb9f-24w22\" (UID: \"637e5d57-19bd-4fa5-b4ef-6c3884a5ff43\") " pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:32.161371 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.161232 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/637e5d57-19bd-4fa5-b4ef-6c3884a5ff43-oauth-serving-cert\") pod \"console-7685ddfb9f-24w22\" (UID: \"637e5d57-19bd-4fa5-b4ef-6c3884a5ff43\") " pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:32.161495 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.161469 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/637e5d57-19bd-4fa5-b4ef-6c3884a5ff43-console-oauth-config\") pod \"console-7685ddfb9f-24w22\" (UID: \"637e5d57-19bd-4fa5-b4ef-6c3884a5ff43\") " pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:32.161554 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.161510 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/637e5d57-19bd-4fa5-b4ef-6c3884a5ff43-service-ca\") pod \"console-7685ddfb9f-24w22\" (UID: \"637e5d57-19bd-4fa5-b4ef-6c3884a5ff43\") " pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:32.161554 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.161542 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/637e5d57-19bd-4fa5-b4ef-6c3884a5ff43-trusted-ca-bundle\") pod \"console-7685ddfb9f-24w22\" (UID: \"637e5d57-19bd-4fa5-b4ef-6c3884a5ff43\") " pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:32.161655 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.161564 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqxzv\" (UniqueName: \"kubernetes.io/projected/637e5d57-19bd-4fa5-b4ef-6c3884a5ff43-kube-api-access-gqxzv\") pod \"console-7685ddfb9f-24w22\" (UID: \"637e5d57-19bd-4fa5-b4ef-6c3884a5ff43\") " pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:32.162077 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.162049 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/637e5d57-19bd-4fa5-b4ef-6c3884a5ff43-oauth-serving-cert\") pod \"console-7685ddfb9f-24w22\" (UID: \"637e5d57-19bd-4fa5-b4ef-6c3884a5ff43\") " pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:32.162218 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.162076 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/637e5d57-19bd-4fa5-b4ef-6c3884a5ff43-console-config\") pod \"console-7685ddfb9f-24w22\" (UID: \"637e5d57-19bd-4fa5-b4ef-6c3884a5ff43\") " pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:32.162218 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.162198 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/637e5d57-19bd-4fa5-b4ef-6c3884a5ff43-service-ca\") pod \"console-7685ddfb9f-24w22\" (UID: \"637e5d57-19bd-4fa5-b4ef-6c3884a5ff43\") " pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:32.162390 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.162371 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/637e5d57-19bd-4fa5-b4ef-6c3884a5ff43-trusted-ca-bundle\") pod \"console-7685ddfb9f-24w22\" (UID: \"637e5d57-19bd-4fa5-b4ef-6c3884a5ff43\") " pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:32.163795 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.163778 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/637e5d57-19bd-4fa5-b4ef-6c3884a5ff43-console-serving-cert\") pod \"console-7685ddfb9f-24w22\" (UID: \"637e5d57-19bd-4fa5-b4ef-6c3884a5ff43\") " pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:32.163963 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.163944 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/637e5d57-19bd-4fa5-b4ef-6c3884a5ff43-console-oauth-config\") pod \"console-7685ddfb9f-24w22\" (UID: \"637e5d57-19bd-4fa5-b4ef-6c3884a5ff43\") " pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:32.170993 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.170972 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqxzv\" (UniqueName: \"kubernetes.io/projected/637e5d57-19bd-4fa5-b4ef-6c3884a5ff43-kube-api-access-gqxzv\") pod \"console-7685ddfb9f-24w22\" (UID: \"637e5d57-19bd-4fa5-b4ef-6c3884a5ff43\") " pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:32.189476 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.189448 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:04:32.304755 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.304680 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:32.580104 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.580049 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7685ddfb9f-24w22"] Sep 11 01:04:32.582320 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:04:32.582292 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod637e5d57_19bd_4fa5_b4ef_6c3884a5ff43.slice/crio-5d03241a7be4f8a0a482fc30f46f6818774538f8d9d30d836842361987fc468d WatchSource:0}: Error finding container 5d03241a7be4f8a0a482fc30f46f6818774538f8d9d30d836842361987fc468d: Status 404 returned error can't find the container with id 5d03241a7be4f8a0a482fc30f46f6818774538f8d9d30d836842361987fc468d Sep 11 01:04:32.779248 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.779216 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:04:32.839438 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:32.839343 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:04:33.291339 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:33.291307 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7685ddfb9f-24w22" event={"ID":"637e5d57-19bd-4fa5-b4ef-6c3884a5ff43","Type":"ContainerStarted","Data":"07ae340636eea752232d823c73962057030c6f4ff60108509fde46747a43f0b0"} Sep 11 01:04:33.291753 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:33.291354 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7685ddfb9f-24w22" event={"ID":"637e5d57-19bd-4fa5-b4ef-6c3884a5ff43","Type":"ContainerStarted","Data":"5d03241a7be4f8a0a482fc30f46f6818774538f8d9d30d836842361987fc468d"} Sep 11 01:04:33.310855 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:33.310807 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7685ddfb9f-24w22" podStartSLOduration=2.310791286 podStartE2EDuration="2.310791286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:04:33.309788875 +0000 UTC m=+400.133242386" watchObservedRunningTime="2026-09-11 01:04:33.310791286 +0000 UTC m=+400.134244796" Sep 11 01:04:40.331056 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:40.331025 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:04:40.440645 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:40.440610 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:04:42.305303 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:42.305263 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:42.305679 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:42.305357 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:42.309896 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:42.309874 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:42.331811 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:42.331789 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7685ddfb9f-24w22" Sep 11 01:04:42.488562 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:42.488531 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:04:42.591061 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:42.590972 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:04:42.666100 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:42.666067 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:04:53.232499 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:53.232454 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:04:54.094343 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:54.094303 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:04:56.306352 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:56.306315 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-wrxwj"] Sep 11 01:04:56.309673 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:56.309648 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-wrxwj" Sep 11 01:04:56.312056 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:56.312029 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 11 01:04:56.312170 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:56.312063 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 11 01:04:56.312170 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:56.312129 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-7wv22\"" Sep 11 01:04:56.319924 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:56.319901 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-wrxwj"] Sep 11 01:04:56.433929 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:56.433901 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7fp9\" (UniqueName: \"kubernetes.io/projected/a24001de-37e3-4482-903d-020d8e9cb18e-kube-api-access-w7fp9\") pod \"authorino-operator-76b478bd78-wrxwj\" (UID: \"a24001de-37e3-4482-903d-020d8e9cb18e\") " pod="kuadrant-system/authorino-operator-76b478bd78-wrxwj" Sep 11 01:04:56.534988 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:56.534962 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7fp9\" (UniqueName: \"kubernetes.io/projected/a24001de-37e3-4482-903d-020d8e9cb18e-kube-api-access-w7fp9\") pod \"authorino-operator-76b478bd78-wrxwj\" (UID: \"a24001de-37e3-4482-903d-020d8e9cb18e\") " pod="kuadrant-system/authorino-operator-76b478bd78-wrxwj" Sep 11 01:04:56.549703 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:56.549675 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7fp9\" (UniqueName: \"kubernetes.io/projected/a24001de-37e3-4482-903d-020d8e9cb18e-kube-api-access-w7fp9\") pod \"authorino-operator-76b478bd78-wrxwj\" (UID: \"a24001de-37e3-4482-903d-020d8e9cb18e\") " pod="kuadrant-system/authorino-operator-76b478bd78-wrxwj" Sep 11 01:04:56.584596 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:56.584520 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:04:56.620533 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:56.620512 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-wrxwj" Sep 11 01:04:56.706347 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:56.706321 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:04:56.897401 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:56.897379 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-wrxwj"] Sep 11 01:04:56.899897 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:04:56.899867 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda24001de_37e3_4482_903d_020d8e9cb18e.slice/crio-68d5690272f3646edea5594e528ebe5c7e25b4cc052db32ff4b1a1cb486a05f7 WatchSource:0}: Error finding container 68d5690272f3646edea5594e528ebe5c7e25b4cc052db32ff4b1a1cb486a05f7: Status 404 returned error can't find the container with id 68d5690272f3646edea5594e528ebe5c7e25b4cc052db32ff4b1a1cb486a05f7 Sep 11 01:04:57.024307 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:57.024274 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:04:57.393948 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:57.393917 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-wrxwj" event={"ID":"a24001de-37e3-4482-903d-020d8e9cb18e","Type":"ContainerStarted","Data":"68d5690272f3646edea5594e528ebe5c7e25b4cc052db32ff4b1a1cb486a05f7"} Sep 11 01:04:57.445162 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:57.445135 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:04:57.829495 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:57.829426 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:04:58.743691 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:58.743640 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:04:58.850752 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:04:58.850716 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:00.410072 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:00.410032 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-wrxwj" event={"ID":"a24001de-37e3-4482-903d-020d8e9cb18e","Type":"ContainerStarted","Data":"b51d532809c4a1a8e2d448794e51e162dc6832c4239d00b0688474ba70d0a603"} Sep 11 01:05:00.410511 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:00.410112 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-wrxwj" Sep 11 01:05:00.452936 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:00.452889 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-wrxwj" podStartSLOduration=1.9423625310000001 podStartE2EDuration="4.45287379s" totalImagesPullingTime="2.510511259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:04:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:05:00.440071444 +0000 UTC m=+427.263524965" watchObservedRunningTime="2026-09-11 01:05:00.45287379 +0000 UTC m=+427.276327300" Sep 11 01:05:00.881664 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:00.881632 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd"] Sep 11 01:05:00.884803 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:00.884787 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" Sep 11 01:05:00.887775 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:00.887754 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sstc2\"" Sep 11 01:05:00.899434 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:00.899408 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd"] Sep 11 01:05:00.973592 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:00.973552 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktq7g\" (UniqueName: \"kubernetes.io/projected/efb9350c-12e4-4a0e-a8ae-987ea4f07111-kube-api-access-ktq7g\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6kkbd\" (UID: \"efb9350c-12e4-4a0e-a8ae-987ea4f07111\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" Sep 11 01:05:00.973745 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:00.973606 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/efb9350c-12e4-4a0e-a8ae-987ea4f07111-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6kkbd\" (UID: \"efb9350c-12e4-4a0e-a8ae-987ea4f07111\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" Sep 11 01:05:01.074133 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:01.074104 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktq7g\" (UniqueName: \"kubernetes.io/projected/efb9350c-12e4-4a0e-a8ae-987ea4f07111-kube-api-access-ktq7g\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6kkbd\" (UID: \"efb9350c-12e4-4a0e-a8ae-987ea4f07111\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" Sep 11 01:05:01.074287 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:01.074140 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/efb9350c-12e4-4a0e-a8ae-987ea4f07111-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6kkbd\" (UID: \"efb9350c-12e4-4a0e-a8ae-987ea4f07111\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" Sep 11 01:05:01.074465 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:01.074444 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/efb9350c-12e4-4a0e-a8ae-987ea4f07111-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6kkbd\" (UID: \"efb9350c-12e4-4a0e-a8ae-987ea4f07111\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" Sep 11 01:05:01.085439 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:01.085419 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktq7g\" (UniqueName: \"kubernetes.io/projected/efb9350c-12e4-4a0e-a8ae-987ea4f07111-kube-api-access-ktq7g\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6kkbd\" (UID: \"efb9350c-12e4-4a0e-a8ae-987ea4f07111\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" Sep 11 01:05:01.194873 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:01.194832 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" Sep 11 01:05:01.228876 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:01.228819 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:01.471468 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:01.471398 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd"] Sep 11 01:05:01.474161 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:01.474123 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:01.475103 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:05:01.475078 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb9350c_12e4_4a0e_a8ae_987ea4f07111.slice/crio-32cdee8b0c7560c2c8325e282260a9736f9621442598591294bdf8d86a0367d6 WatchSource:0}: Error finding container 32cdee8b0c7560c2c8325e282260a9736f9621442598591294bdf8d86a0367d6: Status 404 returned error can't find the container with id 32cdee8b0c7560c2c8325e282260a9736f9621442598591294bdf8d86a0367d6 Sep 11 01:05:02.420114 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:02.420074 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" event={"ID":"efb9350c-12e4-4a0e-a8ae-987ea4f07111","Type":"ContainerStarted","Data":"32cdee8b0c7560c2c8325e282260a9736f9621442598591294bdf8d86a0367d6"} Sep 11 01:05:02.861661 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:02.861577 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl"] Sep 11 01:05:02.866022 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:02.865999 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl" Sep 11 01:05:02.868669 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:02.868643 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-fxs29\"" Sep 11 01:05:02.878916 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:02.878891 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl"] Sep 11 01:05:02.986190 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:02.986164 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvp7w\" (UniqueName: \"kubernetes.io/projected/fadfd70f-d18c-43c0-98d3-034be8701531-kube-api-access-cvp7w\") pod \"limitador-operator-controller-manager-74495c69f8-b6pfl\" (UID: \"fadfd70f-d18c-43c0-98d3-034be8701531\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl" Sep 11 01:05:03.087455 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:03.087415 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvp7w\" (UniqueName: \"kubernetes.io/projected/fadfd70f-d18c-43c0-98d3-034be8701531-kube-api-access-cvp7w\") pod \"limitador-operator-controller-manager-74495c69f8-b6pfl\" (UID: \"fadfd70f-d18c-43c0-98d3-034be8701531\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl" Sep 11 01:05:03.099919 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:03.099888 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvp7w\" (UniqueName: \"kubernetes.io/projected/fadfd70f-d18c-43c0-98d3-034be8701531-kube-api-access-cvp7w\") pod \"limitador-operator-controller-manager-74495c69f8-b6pfl\" (UID: \"fadfd70f-d18c-43c0-98d3-034be8701531\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl" Sep 11 01:05:03.177545 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:03.177519 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl" Sep 11 01:05:03.437563 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:03.437495 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:03.455566 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:03.455545 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl"] Sep 11 01:05:03.457366 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:05:03.457336 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfadfd70f_d18c_43c0_98d3_034be8701531.slice/crio-537894c604043c9cd0f55be1d28705c1bea88fbbdeb7c3afbf29b039dcec8520 WatchSource:0}: Error finding container 537894c604043c9cd0f55be1d28705c1bea88fbbdeb7c3afbf29b039dcec8520: Status 404 returned error can't find the container with id 537894c604043c9cd0f55be1d28705c1bea88fbbdeb7c3afbf29b039dcec8520 Sep 11 01:05:04.085893 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:04.085838 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:04.430009 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:04.429969 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl" event={"ID":"fadfd70f-d18c-43c0-98d3-034be8701531","Type":"ContainerStarted","Data":"537894c604043c9cd0f55be1d28705c1bea88fbbdeb7c3afbf29b039dcec8520"} Sep 11 01:05:05.436940 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:05.436904 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl" event={"ID":"fadfd70f-d18c-43c0-98d3-034be8701531","Type":"ContainerStarted","Data":"66b0d8c98290307395865bbaee43988a8024c36bed16bfd3137d357976d562d7"} Sep 11 01:05:05.437374 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:05.437140 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl" Sep 11 01:05:05.454682 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:05.454636 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl" podStartSLOduration=1.6458995459999999 podStartE2EDuration="3.454617603s" totalImagesPullingTime="1.808718057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:05:05.45361749 +0000 UTC m=+432.277070996" watchObservedRunningTime="2026-09-11 01:05:05.454617603 +0000 UTC m=+432.278071112" Sep 11 01:05:06.450227 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:06.450154 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:06.553147 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:06.553113 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:06.836353 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:06.836256 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:06.957703 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:06.957665 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:07.449399 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:07.449367 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" event={"ID":"efb9350c-12e4-4a0e-a8ae-987ea4f07111","Type":"ContainerStarted","Data":"99d63a92fe812d22800e992128d1caa50ff1df208f85bbcaae1bb88b0dc3dca4"} Sep 11 01:05:07.449708 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:07.449429 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" Sep 11 01:05:07.470109 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:07.470063 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" podStartSLOduration=1.893359537 podStartE2EDuration="7.470047729s" totalImagesPullingTime="5.576688192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:05:07.468622035 +0000 UTC m=+434.292075541" watchObservedRunningTime="2026-09-11 01:05:07.470047729 +0000 UTC m=+434.293501236" Sep 11 01:05:09.522341 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:09.522303 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:09.690492 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:09.690459 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:11.416692 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:11.416663 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-wrxwj" Sep 11 01:05:12.439459 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:12.439426 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:12.541132 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:12.541088 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:12.638601 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:12.638570 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:12.740373 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:12.740307 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:12.840878 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:12.840855 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:12.941060 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:12.941030 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:16.444105 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:16.444028 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl" Sep 11 01:05:18.455697 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:18.455669 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" Sep 11 01:05:19.539643 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:19.539604 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:19.641025 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:19.640992 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:20.661496 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.661461 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd"] Sep 11 01:05:20.661974 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.661652 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" podUID="efb9350c-12e4-4a0e-a8ae-987ea4f07111" containerName="manager" containerID="cri-o://99d63a92fe812d22800e992128d1caa50ff1df208f85bbcaae1bb88b0dc3dca4" gracePeriod=2 Sep 11 01:05:20.663751 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:05:20.663714 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-6kkbd\" is forbidden: User \"system:node:ip-10-0-135-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-107.ec2.internal' and this object" podUID="efb9350c-12e4-4a0e-a8ae-987ea4f07111" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" Sep 11 01:05:20.665031 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.665007 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd"] Sep 11 01:05:20.692422 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.692399 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nnzvg"] Sep 11 01:05:20.692685 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.692674 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb9350c-12e4-4a0e-a8ae-987ea4f07111" containerName="manager" Sep 11 01:05:20.692732 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.692687 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb9350c-12e4-4a0e-a8ae-987ea4f07111" containerName="manager" Sep 11 01:05:20.692768 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.692753 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="efb9350c-12e4-4a0e-a8ae-987ea4f07111" containerName="manager" Sep 11 01:05:20.695501 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.695483 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nnzvg" Sep 11 01:05:20.697764 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:05:20.697735 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-6kkbd\" is forbidden: User \"system:node:ip-10-0-135-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-107.ec2.internal' and this object" podUID="efb9350c-12e4-4a0e-a8ae-987ea4f07111" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" Sep 11 01:05:20.728209 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.728171 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nnzvg"] Sep 11 01:05:20.730222 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.730196 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl"] Sep 11 01:05:20.730390 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.730368 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl" podUID="fadfd70f-d18c-43c0-98d3-034be8701531" containerName="manager" containerID="cri-o://66b0d8c98290307395865bbaee43988a8024c36bed16bfd3137d357976d562d7" gracePeriod=2 Sep 11 01:05:20.739177 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.739152 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl"] Sep 11 01:05:20.746371 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:05:20.746341 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-b6pfl\" is forbidden: User \"system:node:ip-10-0-135-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-107.ec2.internal' and this object" podUID="fadfd70f-d18c-43c0-98d3-034be8701531" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl" Sep 11 01:05:20.748970 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:05:20.748924 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-6kkbd\" is forbidden: User \"system:node:ip-10-0-135-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-107.ec2.internal' and this object" podUID="efb9350c-12e4-4a0e-a8ae-987ea4f07111" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" Sep 11 01:05:20.750998 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.750974 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-jzpth"] Sep 11 01:05:20.751404 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.751389 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fadfd70f-d18c-43c0-98d3-034be8701531" containerName="manager" Sep 11 01:05:20.751479 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.751407 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fadfd70f-d18c-43c0-98d3-034be8701531" containerName="manager" Sep 11 01:05:20.751517 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.751496 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="fadfd70f-d18c-43c0-98d3-034be8701531" containerName="manager" Sep 11 01:05:20.754499 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.754479 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jzpth" Sep 11 01:05:20.757151 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:05:20.757121 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-b6pfl\" is forbidden: User \"system:node:ip-10-0-135-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-107.ec2.internal' and this object" podUID="fadfd70f-d18c-43c0-98d3-034be8701531" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl" Sep 11 01:05:20.779128 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.779104 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-jzpth"] Sep 11 01:05:20.794426 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:05:20.794396 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-6kkbd\" is forbidden: User \"system:node:ip-10-0-135-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-107.ec2.internal' and this object" podUID="efb9350c-12e4-4a0e-a8ae-987ea4f07111" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" Sep 11 01:05:20.821875 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.821838 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29f0de2e-04f4-442c-bf72-fb2bc16c168e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nnzvg\" (UID: \"29f0de2e-04f4-442c-bf72-fb2bc16c168e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nnzvg" Sep 11 01:05:20.821972 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.821954 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk44d\" (UniqueName: \"kubernetes.io/projected/29f0de2e-04f4-442c-bf72-fb2bc16c168e-kube-api-access-qk44d\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nnzvg\" (UID: \"29f0de2e-04f4-442c-bf72-fb2bc16c168e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nnzvg" Sep 11 01:05:20.923480 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.923439 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29f0de2e-04f4-442c-bf72-fb2bc16c168e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nnzvg\" (UID: \"29f0de2e-04f4-442c-bf72-fb2bc16c168e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nnzvg" Sep 11 01:05:20.923594 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.923542 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk44d\" (UniqueName: \"kubernetes.io/projected/29f0de2e-04f4-442c-bf72-fb2bc16c168e-kube-api-access-qk44d\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nnzvg\" (UID: \"29f0de2e-04f4-442c-bf72-fb2bc16c168e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nnzvg" Sep 11 01:05:20.923663 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.923605 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6zs6\" (UniqueName: \"kubernetes.io/projected/c43cefa7-f77a-49b2-9569-4c27b4b5e68c-kube-api-access-d6zs6\") pod \"limitador-operator-controller-manager-74495c69f8-jzpth\" (UID: \"c43cefa7-f77a-49b2-9569-4c27b4b5e68c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jzpth" Sep 11 01:05:20.923933 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.923904 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29f0de2e-04f4-442c-bf72-fb2bc16c168e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nnzvg\" (UID: \"29f0de2e-04f4-442c-bf72-fb2bc16c168e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nnzvg" Sep 11 01:05:20.932017 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.931990 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk44d\" (UniqueName: \"kubernetes.io/projected/29f0de2e-04f4-442c-bf72-fb2bc16c168e-kube-api-access-qk44d\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nnzvg\" (UID: \"29f0de2e-04f4-442c-bf72-fb2bc16c168e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nnzvg" Sep 11 01:05:20.960594 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.960575 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" Sep 11 01:05:20.963127 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:20.963094 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl" Sep 11 01:05:20.963232 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:05:20.963209 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-b6pfl\" is forbidden: User \"system:node:ip-10-0-135-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-107.ec2.internal' and this object" podUID="fadfd70f-d18c-43c0-98d3-034be8701531" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl" Sep 11 01:05:20.965100 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:05:20.965076 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-6kkbd\" is forbidden: User \"system:node:ip-10-0-135-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-107.ec2.internal' and this object" podUID="efb9350c-12e4-4a0e-a8ae-987ea4f07111" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" Sep 11 01:05:20.966865 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:05:20.966824 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-b6pfl\" is forbidden: User \"system:node:ip-10-0-135-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-107.ec2.internal' and this object" podUID="fadfd70f-d18c-43c0-98d3-034be8701531" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl" Sep 11 01:05:20.968554 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:05:20.968534 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-6kkbd\" is forbidden: User \"system:node:ip-10-0-135-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-107.ec2.internal' and this object" podUID="efb9350c-12e4-4a0e-a8ae-987ea4f07111" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" Sep 11 01:05:21.024963 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.024935 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6zs6\" (UniqueName: \"kubernetes.io/projected/c43cefa7-f77a-49b2-9569-4c27b4b5e68c-kube-api-access-d6zs6\") pod \"limitador-operator-controller-manager-74495c69f8-jzpth\" (UID: \"c43cefa7-f77a-49b2-9569-4c27b4b5e68c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jzpth" Sep 11 01:05:21.033074 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.033050 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6zs6\" (UniqueName: \"kubernetes.io/projected/c43cefa7-f77a-49b2-9569-4c27b4b5e68c-kube-api-access-d6zs6\") pod \"limitador-operator-controller-manager-74495c69f8-jzpth\" (UID: \"c43cefa7-f77a-49b2-9569-4c27b4b5e68c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jzpth" Sep 11 01:05:21.038873 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.038852 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nnzvg" Sep 11 01:05:21.106402 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.106374 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jzpth" Sep 11 01:05:21.125889 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.125861 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvp7w\" (UniqueName: \"kubernetes.io/projected/fadfd70f-d18c-43c0-98d3-034be8701531-kube-api-access-cvp7w\") pod \"fadfd70f-d18c-43c0-98d3-034be8701531\" (UID: \"fadfd70f-d18c-43c0-98d3-034be8701531\") " Sep 11 01:05:21.125982 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.125936 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktq7g\" (UniqueName: \"kubernetes.io/projected/efb9350c-12e4-4a0e-a8ae-987ea4f07111-kube-api-access-ktq7g\") pod \"efb9350c-12e4-4a0e-a8ae-987ea4f07111\" (UID: \"efb9350c-12e4-4a0e-a8ae-987ea4f07111\") " Sep 11 01:05:21.125982 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.125964 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/efb9350c-12e4-4a0e-a8ae-987ea4f07111-extensions-socket-volume\") pod \"efb9350c-12e4-4a0e-a8ae-987ea4f07111\" (UID: \"efb9350c-12e4-4a0e-a8ae-987ea4f07111\") " Sep 11 01:05:21.126524 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.126499 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb9350c-12e4-4a0e-a8ae-987ea4f07111-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "efb9350c-12e4-4a0e-a8ae-987ea4f07111" (UID: "efb9350c-12e4-4a0e-a8ae-987ea4f07111"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:05:21.127896 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.127830 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fadfd70f-d18c-43c0-98d3-034be8701531-kube-api-access-cvp7w" (OuterVolumeSpecName: "kube-api-access-cvp7w") pod "fadfd70f-d18c-43c0-98d3-034be8701531" (UID: "fadfd70f-d18c-43c0-98d3-034be8701531"). InnerVolumeSpecName "kube-api-access-cvp7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:05:21.128034 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.128020 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb9350c-12e4-4a0e-a8ae-987ea4f07111-kube-api-access-ktq7g" (OuterVolumeSpecName: "kube-api-access-ktq7g") pod "efb9350c-12e4-4a0e-a8ae-987ea4f07111" (UID: "efb9350c-12e4-4a0e-a8ae-987ea4f07111"). InnerVolumeSpecName "kube-api-access-ktq7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:05:21.227254 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.227173 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvp7w\" (UniqueName: \"kubernetes.io/projected/fadfd70f-d18c-43c0-98d3-034be8701531-kube-api-access-cvp7w\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:05:21.227254 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.227202 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktq7g\" (UniqueName: \"kubernetes.io/projected/efb9350c-12e4-4a0e-a8ae-987ea4f07111-kube-api-access-ktq7g\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:05:21.227254 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.227211 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/efb9350c-12e4-4a0e-a8ae-987ea4f07111-extensions-socket-volume\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:05:21.316702 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.316679 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nnzvg"] Sep 11 01:05:21.319093 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:05:21.319065 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29f0de2e_04f4_442c_bf72_fb2bc16c168e.slice/crio-a7cc9a377744af4807e70d463d4eca48a4e5b4de3572504361b7b4ce3ac89b0b WatchSource:0}: Error finding container a7cc9a377744af4807e70d463d4eca48a4e5b4de3572504361b7b4ce3ac89b0b: Status 404 returned error can't find the container with id a7cc9a377744af4807e70d463d4eca48a4e5b4de3572504361b7b4ce3ac89b0b Sep 11 01:05:21.374787 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.374762 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-jzpth"] Sep 11 01:05:21.377619 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:05:21.377597 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43cefa7_f77a_49b2_9569_4c27b4b5e68c.slice/crio-50dcc02f5ff91b11a0015b83a54e1d1d63c821448e9de7f6c0478717615d16e5 WatchSource:0}: Error finding container 50dcc02f5ff91b11a0015b83a54e1d1d63c821448e9de7f6c0478717615d16e5: Status 404 returned error can't find the container with id 50dcc02f5ff91b11a0015b83a54e1d1d63c821448e9de7f6c0478717615d16e5 Sep 11 01:05:21.520066 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.519987 2554 generic.go:356] "Generic (PLEG): container finished" podID="efb9350c-12e4-4a0e-a8ae-987ea4f07111" containerID="99d63a92fe812d22800e992128d1caa50ff1df208f85bbcaae1bb88b0dc3dca4" exitCode=0 Sep 11 01:05:21.520066 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.520030 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" Sep 11 01:05:21.520275 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.520086 2554 scope.go:117] "RemoveContainer" containerID="99d63a92fe812d22800e992128d1caa50ff1df208f85bbcaae1bb88b0dc3dca4" Sep 11 01:05:21.521976 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.521949 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jzpth" event={"ID":"c43cefa7-f77a-49b2-9569-4c27b4b5e68c","Type":"ContainerStarted","Data":"ef4e19197d280510bb74ab618ce4bdcac9714164e4a924bc434cebfcd3fb924e"} Sep 11 01:05:21.522130 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.522005 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jzpth" Sep 11 01:05:21.522130 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.522027 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jzpth" event={"ID":"c43cefa7-f77a-49b2-9569-4c27b4b5e68c","Type":"ContainerStarted","Data":"50dcc02f5ff91b11a0015b83a54e1d1d63c821448e9de7f6c0478717615d16e5"} Sep 11 01:05:21.522481 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:05:21.522451 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-b6pfl\" is forbidden: User \"system:node:ip-10-0-135-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-107.ec2.internal' and this object" podUID="fadfd70f-d18c-43c0-98d3-034be8701531" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl" Sep 11 01:05:21.523478 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.523427 2554 generic.go:356] "Generic (PLEG): container finished" podID="fadfd70f-d18c-43c0-98d3-034be8701531" containerID="66b0d8c98290307395865bbaee43988a8024c36bed16bfd3137d357976d562d7" exitCode=0 Sep 11 01:05:21.523570 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.523479 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl" Sep 11 01:05:21.524512 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:05:21.524422 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-6kkbd\" is forbidden: User \"system:node:ip-10-0-135-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-107.ec2.internal' and this object" podUID="efb9350c-12e4-4a0e-a8ae-987ea4f07111" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" Sep 11 01:05:21.525478 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.525453 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nnzvg" event={"ID":"29f0de2e-04f4-442c-bf72-fb2bc16c168e","Type":"ContainerStarted","Data":"c4c0ee7a72a136400fa303e48db7aed4d01b808e2991f85ba8aa612d135dc3f0"} Sep 11 01:05:21.525577 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.525501 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nnzvg" event={"ID":"29f0de2e-04f4-442c-bf72-fb2bc16c168e","Type":"ContainerStarted","Data":"a7cc9a377744af4807e70d463d4eca48a4e5b4de3572504361b7b4ce3ac89b0b"} Sep 11 01:05:21.525577 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.525536 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nnzvg" Sep 11 01:05:21.526367 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:05:21.526344 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-b6pfl\" is forbidden: User \"system:node:ip-10-0-135-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-107.ec2.internal' and this object" podUID="fadfd70f-d18c-43c0-98d3-034be8701531" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl" Sep 11 01:05:21.529373 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.529284 2554 scope.go:117] "RemoveContainer" containerID="99d63a92fe812d22800e992128d1caa50ff1df208f85bbcaae1bb88b0dc3dca4" Sep 11 01:05:21.529616 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:05:21.529589 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99d63a92fe812d22800e992128d1caa50ff1df208f85bbcaae1bb88b0dc3dca4\": container with ID starting with 99d63a92fe812d22800e992128d1caa50ff1df208f85bbcaae1bb88b0dc3dca4 not found: ID does not exist" containerID="99d63a92fe812d22800e992128d1caa50ff1df208f85bbcaae1bb88b0dc3dca4" Sep 11 01:05:21.529710 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.529627 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d63a92fe812d22800e992128d1caa50ff1df208f85bbcaae1bb88b0dc3dca4"} err="failed to get container status \"99d63a92fe812d22800e992128d1caa50ff1df208f85bbcaae1bb88b0dc3dca4\": rpc error: code = NotFound desc = could not find container \"99d63a92fe812d22800e992128d1caa50ff1df208f85bbcaae1bb88b0dc3dca4\": container with ID starting with 99d63a92fe812d22800e992128d1caa50ff1df208f85bbcaae1bb88b0dc3dca4 not found: ID does not exist" Sep 11 01:05:21.529710 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.529649 2554 scope.go:117] "RemoveContainer" containerID="66b0d8c98290307395865bbaee43988a8024c36bed16bfd3137d357976d562d7" Sep 11 01:05:21.536809 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.536792 2554 scope.go:117] "RemoveContainer" containerID="66b0d8c98290307395865bbaee43988a8024c36bed16bfd3137d357976d562d7" Sep 11 01:05:21.537111 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:05:21.537086 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66b0d8c98290307395865bbaee43988a8024c36bed16bfd3137d357976d562d7\": container with ID starting with 66b0d8c98290307395865bbaee43988a8024c36bed16bfd3137d357976d562d7 not found: ID does not exist" containerID="66b0d8c98290307395865bbaee43988a8024c36bed16bfd3137d357976d562d7" Sep 11 01:05:21.537184 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.537111 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b0d8c98290307395865bbaee43988a8024c36bed16bfd3137d357976d562d7"} err="failed to get container status \"66b0d8c98290307395865bbaee43988a8024c36bed16bfd3137d357976d562d7\": rpc error: code = NotFound desc = could not find container \"66b0d8c98290307395865bbaee43988a8024c36bed16bfd3137d357976d562d7\": container with ID starting with 66b0d8c98290307395865bbaee43988a8024c36bed16bfd3137d357976d562d7 not found: ID does not exist" Sep 11 01:05:21.546644 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.546603 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jzpth" podStartSLOduration=1.546588898 podStartE2EDuration="1.546588898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:05:21.545200485 +0000 UTC m=+448.368654007" watchObservedRunningTime="2026-09-11 01:05:21.546588898 +0000 UTC m=+448.370042409" Sep 11 01:05:21.547422 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:05:21.547401 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-6kkbd\" is forbidden: User \"system:node:ip-10-0-135-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-107.ec2.internal' and this object" podUID="efb9350c-12e4-4a0e-a8ae-987ea4f07111" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" Sep 11 01:05:21.549350 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:05:21.549325 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-b6pfl\" is forbidden: User \"system:node:ip-10-0-135-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-107.ec2.internal' and this object" podUID="fadfd70f-d18c-43c0-98d3-034be8701531" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-b6pfl" Sep 11 01:05:21.551402 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:05:21.551375 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-6kkbd\" is forbidden: User \"system:node:ip-10-0-135-107.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-107.ec2.internal' and this object" podUID="efb9350c-12e4-4a0e-a8ae-987ea4f07111" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6kkbd" Sep 11 01:05:21.571693 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.571655 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nnzvg" podStartSLOduration=1.571643612 podStartE2EDuration="1.571643612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:05:21.571446189 +0000 UTC m=+448.394899700" watchObservedRunningTime="2026-09-11 01:05:21.571643612 +0000 UTC m=+448.395097122" Sep 11 01:05:21.678435 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.678404 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efb9350c-12e4-4a0e-a8ae-987ea4f07111" path="/var/lib/kubelet/pods/efb9350c-12e4-4a0e-a8ae-987ea4f07111/volumes" Sep 11 01:05:21.678773 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:21.678758 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fadfd70f-d18c-43c0-98d3-034be8701531" path="/var/lib/kubelet/pods/fadfd70f-d18c-43c0-98d3-034be8701531/volumes" Sep 11 01:05:23.719820 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:23.719784 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:23.819383 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:23.819351 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:26.565201 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:26.565170 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:26.665137 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:26.665110 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:26.829874 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:26.829791 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:26.947981 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:26.947952 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:27.042783 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:27.042755 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:32.521763 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:32.521731 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:32.534095 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:32.534069 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-jzpth" Sep 11 01:05:32.534303 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:32.534287 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nnzvg" Sep 11 01:05:32.624307 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:32.624276 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:32.929025 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:32.928988 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:33.023858 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:33.023818 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:33.320288 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:33.320207 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:33.420356 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:33.420325 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:37.069765 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:37.069731 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nnzvg"] Sep 11 01:05:37.070250 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:37.070000 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nnzvg" podUID="29f0de2e-04f4-442c-bf72-fb2bc16c168e" containerName="manager" containerID="cri-o://c4c0ee7a72a136400fa303e48db7aed4d01b808e2991f85ba8aa612d135dc3f0" gracePeriod=10 Sep 11 01:05:37.357088 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:37.357014 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kdjzs"] Sep 11 01:05:37.364390 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:37.364364 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kdjzs" Sep 11 01:05:37.380904 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:37.380883 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kdjzs"] Sep 11 01:05:37.443016 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:37.442987 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b75f844c-c91a-485a-82db-420603cdbbae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kdjzs\" (UID: \"b75f844c-c91a-485a-82db-420603cdbbae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kdjzs" Sep 11 01:05:37.443128 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:37.443107 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgx86\" (UniqueName: \"kubernetes.io/projected/b75f844c-c91a-485a-82db-420603cdbbae-kube-api-access-zgx86\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kdjzs\" (UID: \"b75f844c-c91a-485a-82db-420603cdbbae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kdjzs" Sep 11 01:05:37.543919 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:37.543880 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b75f844c-c91a-485a-82db-420603cdbbae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kdjzs\" (UID: \"b75f844c-c91a-485a-82db-420603cdbbae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kdjzs" Sep 11 01:05:37.544019 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:37.543957 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgx86\" (UniqueName: \"kubernetes.io/projected/b75f844c-c91a-485a-82db-420603cdbbae-kube-api-access-zgx86\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kdjzs\" (UID: \"b75f844c-c91a-485a-82db-420603cdbbae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kdjzs" Sep 11 01:05:37.544280 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:37.544261 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b75f844c-c91a-485a-82db-420603cdbbae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kdjzs\" (UID: \"b75f844c-c91a-485a-82db-420603cdbbae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kdjzs" Sep 11 01:05:37.553054 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:37.553033 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgx86\" (UniqueName: \"kubernetes.io/projected/b75f844c-c91a-485a-82db-420603cdbbae-kube-api-access-zgx86\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kdjzs\" (UID: \"b75f844c-c91a-485a-82db-420603cdbbae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kdjzs" Sep 11 01:05:37.674217 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:37.674193 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kdjzs" Sep 11 01:05:37.835254 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:37.835232 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nnzvg" Sep 11 01:05:37.946688 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:37.946660 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qk44d\" (UniqueName: \"kubernetes.io/projected/29f0de2e-04f4-442c-bf72-fb2bc16c168e-kube-api-access-qk44d\") pod \"29f0de2e-04f4-442c-bf72-fb2bc16c168e\" (UID: \"29f0de2e-04f4-442c-bf72-fb2bc16c168e\") " Sep 11 01:05:37.946822 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:37.946753 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29f0de2e-04f4-442c-bf72-fb2bc16c168e-extensions-socket-volume\") pod \"29f0de2e-04f4-442c-bf72-fb2bc16c168e\" (UID: \"29f0de2e-04f4-442c-bf72-fb2bc16c168e\") " Sep 11 01:05:37.947142 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:37.947116 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f0de2e-04f4-442c-bf72-fb2bc16c168e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "29f0de2e-04f4-442c-bf72-fb2bc16c168e" (UID: "29f0de2e-04f4-442c-bf72-fb2bc16c168e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:05:37.948721 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:37.948705 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29f0de2e-04f4-442c-bf72-fb2bc16c168e-kube-api-access-qk44d" (OuterVolumeSpecName: "kube-api-access-qk44d") pod "29f0de2e-04f4-442c-bf72-fb2bc16c168e" (UID: "29f0de2e-04f4-442c-bf72-fb2bc16c168e"). InnerVolumeSpecName "kube-api-access-qk44d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:05:37.953585 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:37.953566 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kdjzs"] Sep 11 01:05:37.955862 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:05:37.955828 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75f844c_c91a_485a_82db_420603cdbbae.slice/crio-babbe9989442d35e564c2adeee1af51eb71869b9618e0ac99b066cef021145ea WatchSource:0}: Error finding container babbe9989442d35e564c2adeee1af51eb71869b9618e0ac99b066cef021145ea: Status 404 returned error can't find the container with id babbe9989442d35e564c2adeee1af51eb71869b9618e0ac99b066cef021145ea Sep 11 01:05:38.047407 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:38.047384 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29f0de2e-04f4-442c-bf72-fb2bc16c168e-extensions-socket-volume\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:05:38.047407 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:38.047407 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk44d\" (UniqueName: \"kubernetes.io/projected/29f0de2e-04f4-442c-bf72-fb2bc16c168e-kube-api-access-qk44d\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:05:38.605958 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:38.605920 2554 generic.go:356] "Generic (PLEG): container finished" podID="29f0de2e-04f4-442c-bf72-fb2bc16c168e" containerID="c4c0ee7a72a136400fa303e48db7aed4d01b808e2991f85ba8aa612d135dc3f0" exitCode=0 Sep 11 01:05:38.606360 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:38.605966 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nnzvg" Sep 11 01:05:38.606360 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:38.606007 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nnzvg" event={"ID":"29f0de2e-04f4-442c-bf72-fb2bc16c168e","Type":"ContainerDied","Data":"c4c0ee7a72a136400fa303e48db7aed4d01b808e2991f85ba8aa612d135dc3f0"} Sep 11 01:05:38.606360 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:38.606057 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nnzvg" event={"ID":"29f0de2e-04f4-442c-bf72-fb2bc16c168e","Type":"ContainerDied","Data":"a7cc9a377744af4807e70d463d4eca48a4e5b4de3572504361b7b4ce3ac89b0b"} Sep 11 01:05:38.606360 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:38.606089 2554 scope.go:117] "RemoveContainer" containerID="c4c0ee7a72a136400fa303e48db7aed4d01b808e2991f85ba8aa612d135dc3f0" Sep 11 01:05:38.607977 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:38.607951 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kdjzs" event={"ID":"b75f844c-c91a-485a-82db-420603cdbbae","Type":"ContainerStarted","Data":"f723ffa3196d38165a0eb0b51723a4f992ed430cb80b49ba9f73bf2a1ee5d2b2"} Sep 11 01:05:38.608082 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:38.608000 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kdjzs" event={"ID":"b75f844c-c91a-485a-82db-420603cdbbae","Type":"ContainerStarted","Data":"babbe9989442d35e564c2adeee1af51eb71869b9618e0ac99b066cef021145ea"} Sep 11 01:05:38.608082 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:38.608031 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kdjzs" Sep 11 01:05:38.614689 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:38.614672 2554 scope.go:117] "RemoveContainer" containerID="c4c0ee7a72a136400fa303e48db7aed4d01b808e2991f85ba8aa612d135dc3f0" Sep 11 01:05:38.614951 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:05:38.614934 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4c0ee7a72a136400fa303e48db7aed4d01b808e2991f85ba8aa612d135dc3f0\": container with ID starting with c4c0ee7a72a136400fa303e48db7aed4d01b808e2991f85ba8aa612d135dc3f0 not found: ID does not exist" containerID="c4c0ee7a72a136400fa303e48db7aed4d01b808e2991f85ba8aa612d135dc3f0" Sep 11 01:05:38.615005 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:38.614960 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4c0ee7a72a136400fa303e48db7aed4d01b808e2991f85ba8aa612d135dc3f0"} err="failed to get container status \"c4c0ee7a72a136400fa303e48db7aed4d01b808e2991f85ba8aa612d135dc3f0\": rpc error: code = NotFound desc = could not find container \"c4c0ee7a72a136400fa303e48db7aed4d01b808e2991f85ba8aa612d135dc3f0\": container with ID starting with c4c0ee7a72a136400fa303e48db7aed4d01b808e2991f85ba8aa612d135dc3f0 not found: ID does not exist" Sep 11 01:05:38.629217 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:38.629175 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kdjzs" podStartSLOduration=1.629162672 podStartE2EDuration="1.629162672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:05:38.629108558 +0000 UTC m=+465.452562071" watchObservedRunningTime="2026-09-11 01:05:38.629162672 +0000 UTC m=+465.452616183" Sep 11 01:05:38.644173 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:38.644150 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nnzvg"] Sep 11 01:05:38.649809 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:38.649787 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nnzvg"] Sep 11 01:05:39.538053 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:39.538019 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:39.645534 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:39.645502 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:39.677542 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:39.677515 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29f0de2e-04f4-442c-bf72-fb2bc16c168e" path="/var/lib/kubelet/pods/29f0de2e-04f4-442c-bf72-fb2bc16c168e/volumes" Sep 11 01:05:41.970757 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:41.970725 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:42.071030 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:42.070997 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:49.621358 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:49.621331 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kdjzs" Sep 11 01:05:53.453035 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:53.452991 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:53.680594 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:53.680567 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:58.182562 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:58.182526 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:58.187459 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:58.187433 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-w4xr8"] Sep 11 01:05:58.187746 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:58.187733 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29f0de2e-04f4-442c-bf72-fb2bc16c168e" containerName="manager" Sep 11 01:05:58.187786 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:58.187747 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="29f0de2e-04f4-442c-bf72-fb2bc16c168e" containerName="manager" Sep 11 01:05:58.187830 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:58.187820 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="29f0de2e-04f4-442c-bf72-fb2bc16c168e" containerName="manager" Sep 11 01:05:58.190711 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:58.190689 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-w4xr8" Sep 11 01:05:58.193805 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:58.193773 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 11 01:05:58.194365 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:58.194351 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-txtlm\"" Sep 11 01:05:58.199571 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:58.199549 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-w4xr8"] Sep 11 01:05:58.286339 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:58.286299 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:58.286685 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:58.286663 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-w4xr8"] Sep 11 01:05:58.294293 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:58.294269 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd9fd\" (UniqueName: \"kubernetes.io/projected/3887dd97-20c3-4776-b29d-44548c2b6397-kube-api-access-sd9fd\") pod \"limitador-limitador-8588749977-w4xr8\" (UID: \"3887dd97-20c3-4776-b29d-44548c2b6397\") " pod="kuadrant-system/limitador-limitador-8588749977-w4xr8" Sep 11 01:05:58.294410 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:58.294300 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3887dd97-20c3-4776-b29d-44548c2b6397-config-file\") pod \"limitador-limitador-8588749977-w4xr8\" (UID: \"3887dd97-20c3-4776-b29d-44548c2b6397\") " pod="kuadrant-system/limitador-limitador-8588749977-w4xr8" Sep 11 01:05:58.385141 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:58.385112 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:58.395181 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:58.395146 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd9fd\" (UniqueName: \"kubernetes.io/projected/3887dd97-20c3-4776-b29d-44548c2b6397-kube-api-access-sd9fd\") pod \"limitador-limitador-8588749977-w4xr8\" (UID: \"3887dd97-20c3-4776-b29d-44548c2b6397\") " pod="kuadrant-system/limitador-limitador-8588749977-w4xr8" Sep 11 01:05:58.395312 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:58.395192 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3887dd97-20c3-4776-b29d-44548c2b6397-config-file\") pod \"limitador-limitador-8588749977-w4xr8\" (UID: \"3887dd97-20c3-4776-b29d-44548c2b6397\") " pod="kuadrant-system/limitador-limitador-8588749977-w4xr8" Sep 11 01:05:58.395870 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:58.395836 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3887dd97-20c3-4776-b29d-44548c2b6397-config-file\") pod \"limitador-limitador-8588749977-w4xr8\" (UID: \"3887dd97-20c3-4776-b29d-44548c2b6397\") " pod="kuadrant-system/limitador-limitador-8588749977-w4xr8" Sep 11 01:05:58.403691 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:58.403661 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd9fd\" (UniqueName: \"kubernetes.io/projected/3887dd97-20c3-4776-b29d-44548c2b6397-kube-api-access-sd9fd\") pod \"limitador-limitador-8588749977-w4xr8\" (UID: \"3887dd97-20c3-4776-b29d-44548c2b6397\") " pod="kuadrant-system/limitador-limitador-8588749977-w4xr8" Sep 11 01:05:58.501944 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:58.501885 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-w4xr8" Sep 11 01:05:58.773211 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:58.773181 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-w4xr8"] Sep 11 01:05:58.775595 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:05:58.775573 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3887dd97_20c3_4776_b29d_44548c2b6397.slice/crio-79662d1619a8a019d50a8397091970503faa4dac7ca20295590f42e69bb55648 WatchSource:0}: Error finding container 79662d1619a8a019d50a8397091970503faa4dac7ca20295590f42e69bb55648: Status 404 returned error can't find the container with id 79662d1619a8a019d50a8397091970503faa4dac7ca20295590f42e69bb55648 Sep 11 01:05:59.044788 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:59.044711 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:59.147195 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:59.147144 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:05:59.489210 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:59.489175 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-66dfc"] Sep 11 01:05:59.491873 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:59.491856 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-66dfc" Sep 11 01:05:59.495312 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:59.495290 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bjhvj\"" Sep 11 01:05:59.500786 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:59.500759 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-66dfc"] Sep 11 01:05:59.605686 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:59.605660 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfrkl\" (UniqueName: \"kubernetes.io/projected/15335aa6-03ba-49ad-9de8-5d99323c6fb4-kube-api-access-gfrkl\") pod \"authorino-59c974bd48-66dfc\" (UID: \"15335aa6-03ba-49ad-9de8-5d99323c6fb4\") " pod="kuadrant-system/authorino-59c974bd48-66dfc" Sep 11 01:05:59.703283 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:59.703252 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-w4xr8" event={"ID":"3887dd97-20c3-4776-b29d-44548c2b6397","Type":"ContainerStarted","Data":"79662d1619a8a019d50a8397091970503faa4dac7ca20295590f42e69bb55648"} Sep 11 01:05:59.706517 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:59.706497 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfrkl\" (UniqueName: \"kubernetes.io/projected/15335aa6-03ba-49ad-9de8-5d99323c6fb4-kube-api-access-gfrkl\") pod \"authorino-59c974bd48-66dfc\" (UID: \"15335aa6-03ba-49ad-9de8-5d99323c6fb4\") " pod="kuadrant-system/authorino-59c974bd48-66dfc" Sep 11 01:05:59.715642 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:59.715621 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfrkl\" (UniqueName: \"kubernetes.io/projected/15335aa6-03ba-49ad-9de8-5d99323c6fb4-kube-api-access-gfrkl\") pod \"authorino-59c974bd48-66dfc\" (UID: \"15335aa6-03ba-49ad-9de8-5d99323c6fb4\") " pod="kuadrant-system/authorino-59c974bd48-66dfc" Sep 11 01:05:59.801599 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:05:59.801542 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-66dfc" Sep 11 01:06:00.066025 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:00.065959 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-66dfc"] Sep 11 01:06:00.068798 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:06:00.068773 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15335aa6_03ba_49ad_9de8_5d99323c6fb4.slice/crio-00963edfde5bb0c85e9873f3459ad8a105d1630e46a5dc9662aa988653682dfc WatchSource:0}: Error finding container 00963edfde5bb0c85e9873f3459ad8a105d1630e46a5dc9662aa988653682dfc: Status 404 returned error can't find the container with id 00963edfde5bb0c85e9873f3459ad8a105d1630e46a5dc9662aa988653682dfc Sep 11 01:06:00.709570 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:00.709530 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-w4xr8" event={"ID":"3887dd97-20c3-4776-b29d-44548c2b6397","Type":"ContainerStarted","Data":"0035c67758281e7065c398cb94cb5efe764cb6fcb0a091826e20512678cbfad3"} Sep 11 01:06:00.710033 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:00.709650 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-w4xr8" Sep 11 01:06:00.711101 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:00.711078 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-66dfc" event={"ID":"15335aa6-03ba-49ad-9de8-5d99323c6fb4","Type":"ContainerStarted","Data":"00963edfde5bb0c85e9873f3459ad8a105d1630e46a5dc9662aa988653682dfc"} Sep 11 01:06:00.734521 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:00.734472 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-w4xr8" podStartSLOduration=0.919053292 podStartE2EDuration="2.734452241s" totalImagesPullingTime="1.815398949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:05:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:06:00.732989285 +0000 UTC m=+487.556442797" watchObservedRunningTime="2026-09-11 01:06:00.734452241 +0000 UTC m=+487.557905753" Sep 11 01:06:01.413984 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:01.413953 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:06:01.717241 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:01.717204 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-66dfc" event={"ID":"15335aa6-03ba-49ad-9de8-5d99323c6fb4","Type":"ContainerStarted","Data":"d1c94677ad64bdd3184d4737115f53621baff93462184ac50194e0481c4218ba"} Sep 11 01:06:01.733673 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:01.733627 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-66dfc" podStartSLOduration=1.235792716 podStartE2EDuration="2.733611699s" totalImagesPullingTime="1.497818983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:05:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:06:01.732106248 +0000 UTC m=+488.555559760" watchObservedRunningTime="2026-09-11 01:06:01.733611699 +0000 UTC m=+488.557065208" Sep 11 01:06:11.718429 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:11.718394 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-w4xr8" Sep 11 01:06:12.249183 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:12.249151 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-w4xr8"] Sep 11 01:06:12.249384 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:12.249316 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-w4xr8" podUID="3887dd97-20c3-4776-b29d-44548c2b6397" containerName="limitador" containerID="cri-o://0035c67758281e7065c398cb94cb5efe764cb6fcb0a091826e20512678cbfad3" gracePeriod=30 Sep 11 01:06:12.767685 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:12.767655 2554 generic.go:356] "Generic (PLEG): container finished" podID="3887dd97-20c3-4776-b29d-44548c2b6397" containerID="0035c67758281e7065c398cb94cb5efe764cb6fcb0a091826e20512678cbfad3" exitCode=0 Sep 11 01:06:12.768018 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:12.767734 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-w4xr8" event={"ID":"3887dd97-20c3-4776-b29d-44548c2b6397","Type":"ContainerDied","Data":"0035c67758281e7065c398cb94cb5efe764cb6fcb0a091826e20512678cbfad3"} Sep 11 01:06:12.797980 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:12.797958 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-w4xr8" Sep 11 01:06:12.901206 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:12.901152 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3887dd97-20c3-4776-b29d-44548c2b6397-config-file\") pod \"3887dd97-20c3-4776-b29d-44548c2b6397\" (UID: \"3887dd97-20c3-4776-b29d-44548c2b6397\") " Sep 11 01:06:12.901206 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:12.901198 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd9fd\" (UniqueName: \"kubernetes.io/projected/3887dd97-20c3-4776-b29d-44548c2b6397-kube-api-access-sd9fd\") pod \"3887dd97-20c3-4776-b29d-44548c2b6397\" (UID: \"3887dd97-20c3-4776-b29d-44548c2b6397\") " Sep 11 01:06:12.901473 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:12.901451 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3887dd97-20c3-4776-b29d-44548c2b6397-config-file" (OuterVolumeSpecName: "config-file") pod "3887dd97-20c3-4776-b29d-44548c2b6397" (UID: "3887dd97-20c3-4776-b29d-44548c2b6397"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:06:12.903178 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:12.903154 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3887dd97-20c3-4776-b29d-44548c2b6397-kube-api-access-sd9fd" (OuterVolumeSpecName: "kube-api-access-sd9fd") pod "3887dd97-20c3-4776-b29d-44548c2b6397" (UID: "3887dd97-20c3-4776-b29d-44548c2b6397"). InnerVolumeSpecName "kube-api-access-sd9fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:06:13.001575 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:13.001551 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd9fd\" (UniqueName: \"kubernetes.io/projected/3887dd97-20c3-4776-b29d-44548c2b6397-kube-api-access-sd9fd\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:06:13.001575 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:13.001571 2554 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3887dd97-20c3-4776-b29d-44548c2b6397-config-file\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:06:13.773937 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:13.773910 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-w4xr8" Sep 11 01:06:13.774271 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:13.773935 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-w4xr8" event={"ID":"3887dd97-20c3-4776-b29d-44548c2b6397","Type":"ContainerDied","Data":"79662d1619a8a019d50a8397091970503faa4dac7ca20295590f42e69bb55648"} Sep 11 01:06:13.774271 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:13.773992 2554 scope.go:117] "RemoveContainer" containerID="0035c67758281e7065c398cb94cb5efe764cb6fcb0a091826e20512678cbfad3" Sep 11 01:06:13.789700 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:13.789678 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-w4xr8"] Sep 11 01:06:13.795292 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:13.795272 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-w4xr8"] Sep 11 01:06:14.352937 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:14.352906 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:06:14.441695 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:14.441662 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:06:14.864094 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:14.864067 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-djjsb"] Sep 11 01:06:14.865193 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:14.865165 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3887dd97-20c3-4776-b29d-44548c2b6397" containerName="limitador" Sep 11 01:06:14.865304 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:14.865197 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3887dd97-20c3-4776-b29d-44548c2b6397" containerName="limitador" Sep 11 01:06:14.865387 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:14.865375 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3887dd97-20c3-4776-b29d-44548c2b6397" containerName="limitador" Sep 11 01:06:14.870686 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:14.870663 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-djjsb" Sep 11 01:06:14.873416 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:14.873394 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-kjh45\"" Sep 11 01:06:14.873515 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:14.873442 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 11 01:06:14.874172 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:14.874159 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 01:06:14.874258 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:14.874211 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 01:06:14.876430 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:14.876412 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-djjsb"] Sep 11 01:06:14.890119 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:14.890093 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:06:15.014979 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:15.014958 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c51d617e-2ed1-40d1-806e-f3cfb0e851e5-data\") pod \"postgres-69fb4594cf-djjsb\" (UID: \"c51d617e-2ed1-40d1-806e-f3cfb0e851e5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-djjsb" Sep 11 01:06:15.015062 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:15.014994 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng426\" (UniqueName: \"kubernetes.io/projected/c51d617e-2ed1-40d1-806e-f3cfb0e851e5-kube-api-access-ng426\") pod \"postgres-69fb4594cf-djjsb\" (UID: \"c51d617e-2ed1-40d1-806e-f3cfb0e851e5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-djjsb" Sep 11 01:06:15.116044 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:15.115994 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c51d617e-2ed1-40d1-806e-f3cfb0e851e5-data\") pod \"postgres-69fb4594cf-djjsb\" (UID: \"c51d617e-2ed1-40d1-806e-f3cfb0e851e5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-djjsb" Sep 11 01:06:15.116044 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:15.116028 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng426\" (UniqueName: \"kubernetes.io/projected/c51d617e-2ed1-40d1-806e-f3cfb0e851e5-kube-api-access-ng426\") pod \"postgres-69fb4594cf-djjsb\" (UID: \"c51d617e-2ed1-40d1-806e-f3cfb0e851e5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-djjsb" Sep 11 01:06:15.116348 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:15.116330 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c51d617e-2ed1-40d1-806e-f3cfb0e851e5-data\") pod \"postgres-69fb4594cf-djjsb\" (UID: \"c51d617e-2ed1-40d1-806e-f3cfb0e851e5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-djjsb" Sep 11 01:06:15.125872 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:15.125834 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng426\" (UniqueName: \"kubernetes.io/projected/c51d617e-2ed1-40d1-806e-f3cfb0e851e5-kube-api-access-ng426\") pod \"postgres-69fb4594cf-djjsb\" (UID: \"c51d617e-2ed1-40d1-806e-f3cfb0e851e5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-djjsb" Sep 11 01:06:15.181799 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:15.181777 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-djjsb" Sep 11 01:06:15.451992 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:15.451966 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-djjsb"] Sep 11 01:06:15.454637 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:06:15.454611 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51d617e_2ed1_40d1_806e_f3cfb0e851e5.slice/crio-ab8e44e320b04430f4646f0e697e545e765f81fab5fac11f47078cd7d1379da5 WatchSource:0}: Error finding container ab8e44e320b04430f4646f0e697e545e765f81fab5fac11f47078cd7d1379da5: Status 404 returned error can't find the container with id ab8e44e320b04430f4646f0e697e545e765f81fab5fac11f47078cd7d1379da5 Sep 11 01:06:15.677740 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:15.677701 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3887dd97-20c3-4776-b29d-44548c2b6397" path="/var/lib/kubelet/pods/3887dd97-20c3-4776-b29d-44548c2b6397/volumes" Sep 11 01:06:15.784483 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:15.784425 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-djjsb" event={"ID":"c51d617e-2ed1-40d1-806e-f3cfb0e851e5","Type":"ContainerStarted","Data":"ab8e44e320b04430f4646f0e697e545e765f81fab5fac11f47078cd7d1379da5"} Sep 11 01:06:20.812920 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:20.812882 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-djjsb" event={"ID":"c51d617e-2ed1-40d1-806e-f3cfb0e851e5","Type":"ContainerStarted","Data":"9486b999022086a7fa73ce4772729d279e1b231b60cb01db9fef0b673f1e5aca"} Sep 11 01:06:20.813277 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:20.812960 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-djjsb" Sep 11 01:06:20.829990 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:20.829947 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-djjsb" podStartSLOduration=2.124808672 podStartE2EDuration="6.829935323s" totalImagesPullingTime="4.705126651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:06:20.82854378 +0000 UTC m=+507.651997290" watchObservedRunningTime="2026-09-11 01:06:20.829935323 +0000 UTC m=+507.653388870" Sep 11 01:06:26.846025 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:26.845997 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-djjsb" Sep 11 01:06:27.170274 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:27.170245 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:06:27.419514 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:27.419478 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:06:29.382251 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:29.382221 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-66dfc"] Sep 11 01:06:29.382642 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:29.382384 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-66dfc" podUID="15335aa6-03ba-49ad-9de8-5d99323c6fb4" containerName="authorino" containerID="cri-o://d1c94677ad64bdd3184d4737115f53621baff93462184ac50194e0481c4218ba" gracePeriod=30 Sep 11 01:06:29.854302 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:29.854265 2554 generic.go:356] "Generic (PLEG): container finished" podID="15335aa6-03ba-49ad-9de8-5d99323c6fb4" containerID="d1c94677ad64bdd3184d4737115f53621baff93462184ac50194e0481c4218ba" exitCode=0 Sep 11 01:06:29.854460 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:29.854303 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-66dfc" event={"ID":"15335aa6-03ba-49ad-9de8-5d99323c6fb4","Type":"ContainerDied","Data":"d1c94677ad64bdd3184d4737115f53621baff93462184ac50194e0481c4218ba"} Sep 11 01:06:30.177189 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:30.177162 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-66dfc" Sep 11 01:06:30.333885 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:30.333861 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfrkl\" (UniqueName: \"kubernetes.io/projected/15335aa6-03ba-49ad-9de8-5d99323c6fb4-kube-api-access-gfrkl\") pod \"15335aa6-03ba-49ad-9de8-5d99323c6fb4\" (UID: \"15335aa6-03ba-49ad-9de8-5d99323c6fb4\") " Sep 11 01:06:30.335855 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:30.335821 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15335aa6-03ba-49ad-9de8-5d99323c6fb4-kube-api-access-gfrkl" (OuterVolumeSpecName: "kube-api-access-gfrkl") pod "15335aa6-03ba-49ad-9de8-5d99323c6fb4" (UID: "15335aa6-03ba-49ad-9de8-5d99323c6fb4"). InnerVolumeSpecName "kube-api-access-gfrkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:06:30.435342 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:30.435318 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfrkl\" (UniqueName: \"kubernetes.io/projected/15335aa6-03ba-49ad-9de8-5d99323c6fb4-kube-api-access-gfrkl\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:06:30.860497 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:30.860426 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-66dfc" event={"ID":"15335aa6-03ba-49ad-9de8-5d99323c6fb4","Type":"ContainerDied","Data":"00963edfde5bb0c85e9873f3459ad8a105d1630e46a5dc9662aa988653682dfc"} Sep 11 01:06:30.860497 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:30.860449 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-66dfc" Sep 11 01:06:30.860497 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:30.860471 2554 scope.go:117] "RemoveContainer" containerID="d1c94677ad64bdd3184d4737115f53621baff93462184ac50194e0481c4218ba" Sep 11 01:06:30.883044 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:30.883023 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-66dfc"] Sep 11 01:06:30.888620 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:30.888590 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-66dfc"] Sep 11 01:06:31.677609 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:31.677574 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15335aa6-03ba-49ad-9de8-5d99323c6fb4" path="/var/lib/kubelet/pods/15335aa6-03ba-49ad-9de8-5d99323c6fb4/volumes" Sep 11 01:06:34.362024 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:34.361991 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:06:34.439585 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:34.439549 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:06:34.503691 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:34.503647 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:06:34.565796 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:34.565768 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8f8d984c8-dbdv8"] Sep 11 01:06:34.566095 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:34.566080 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15335aa6-03ba-49ad-9de8-5d99323c6fb4" containerName="authorino" Sep 11 01:06:34.566095 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:34.566094 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="15335aa6-03ba-49ad-9de8-5d99323c6fb4" containerName="authorino" Sep 11 01:06:34.566179 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:34.566155 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="15335aa6-03ba-49ad-9de8-5d99323c6fb4" containerName="authorino" Sep 11 01:06:34.568577 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:34.568559 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8f8d984c8-dbdv8" Sep 11 01:06:34.571026 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:34.571003 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 11 01:06:34.571403 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:34.571371 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 11 01:06:34.571403 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:34.571394 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 11 01:06:34.571938 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:34.571923 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wp4dg\"" Sep 11 01:06:34.581446 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:34.581425 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8f8d984c8-dbdv8"] Sep 11 01:06:34.666109 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:34.666054 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68jgw\" (UniqueName: \"kubernetes.io/projected/b40367b4-cbb4-4bed-ab22-27752879bbd0-kube-api-access-68jgw\") pod \"maas-controller-8f8d984c8-dbdv8\" (UID: \"b40367b4-cbb4-4bed-ab22-27752879bbd0\") " pod="opendatahub/maas-controller-8f8d984c8-dbdv8" Sep 11 01:06:34.666225 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:34.666113 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b40367b4-cbb4-4bed-ab22-27752879bbd0-metrics-certs\") pod \"maas-controller-8f8d984c8-dbdv8\" (UID: \"b40367b4-cbb4-4bed-ab22-27752879bbd0\") " pod="opendatahub/maas-controller-8f8d984c8-dbdv8" Sep 11 01:06:34.666225 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:34.666207 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b40367b4-cbb4-4bed-ab22-27752879bbd0-webhook-cert\") pod \"maas-controller-8f8d984c8-dbdv8\" (UID: \"b40367b4-cbb4-4bed-ab22-27752879bbd0\") " pod="opendatahub/maas-controller-8f8d984c8-dbdv8" Sep 11 01:06:34.766946 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:34.766921 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b40367b4-cbb4-4bed-ab22-27752879bbd0-webhook-cert\") pod \"maas-controller-8f8d984c8-dbdv8\" (UID: \"b40367b4-cbb4-4bed-ab22-27752879bbd0\") " pod="opendatahub/maas-controller-8f8d984c8-dbdv8" Sep 11 01:06:34.767050 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:34.766977 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68jgw\" (UniqueName: \"kubernetes.io/projected/b40367b4-cbb4-4bed-ab22-27752879bbd0-kube-api-access-68jgw\") pod \"maas-controller-8f8d984c8-dbdv8\" (UID: \"b40367b4-cbb4-4bed-ab22-27752879bbd0\") " pod="opendatahub/maas-controller-8f8d984c8-dbdv8" Sep 11 01:06:34.767050 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:34.767022 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b40367b4-cbb4-4bed-ab22-27752879bbd0-metrics-certs\") pod \"maas-controller-8f8d984c8-dbdv8\" (UID: \"b40367b4-cbb4-4bed-ab22-27752879bbd0\") " pod="opendatahub/maas-controller-8f8d984c8-dbdv8" Sep 11 01:06:34.767163 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:06:34.767068 2554 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 11 01:06:34.767163 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:06:34.767133 2554 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Sep 11 01:06:34.767268 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:06:34.767135 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b40367b4-cbb4-4bed-ab22-27752879bbd0-webhook-cert podName:b40367b4-cbb4-4bed-ab22-27752879bbd0 nodeName:}" failed. No retries permitted until 2026-09-11 01:06:35.267116908 +0000 UTC m=+522.090570420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/b40367b4-cbb4-4bed-ab22-27752879bbd0-webhook-cert") pod "maas-controller-8f8d984c8-dbdv8" (UID: "b40367b4-cbb4-4bed-ab22-27752879bbd0") : secret "maas-controller-webhook-cert" not found Sep 11 01:06:34.767268 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:06:34.767188 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b40367b4-cbb4-4bed-ab22-27752879bbd0-metrics-certs podName:b40367b4-cbb4-4bed-ab22-27752879bbd0 nodeName:}" failed. No retries permitted until 2026-09-11 01:06:35.267171907 +0000 UTC m=+522.090625399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b40367b4-cbb4-4bed-ab22-27752879bbd0-metrics-certs") pod "maas-controller-8f8d984c8-dbdv8" (UID: "b40367b4-cbb4-4bed-ab22-27752879bbd0") : secret "maas-controller-metrics-tls" not found Sep 11 01:06:34.775555 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:34.775530 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68jgw\" (UniqueName: \"kubernetes.io/projected/b40367b4-cbb4-4bed-ab22-27752879bbd0-kube-api-access-68jgw\") pod \"maas-controller-8f8d984c8-dbdv8\" (UID: \"b40367b4-cbb4-4bed-ab22-27752879bbd0\") " pod="opendatahub/maas-controller-8f8d984c8-dbdv8" Sep 11 01:06:34.910740 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:34.910701 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:06:35.049498 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:35.049465 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:06:35.270764 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:35.270734 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b40367b4-cbb4-4bed-ab22-27752879bbd0-webhook-cert\") pod \"maas-controller-8f8d984c8-dbdv8\" (UID: \"b40367b4-cbb4-4bed-ab22-27752879bbd0\") " pod="opendatahub/maas-controller-8f8d984c8-dbdv8" Sep 11 01:06:35.270911 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:35.270796 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b40367b4-cbb4-4bed-ab22-27752879bbd0-metrics-certs\") pod \"maas-controller-8f8d984c8-dbdv8\" (UID: \"b40367b4-cbb4-4bed-ab22-27752879bbd0\") " pod="opendatahub/maas-controller-8f8d984c8-dbdv8" Sep 11 01:06:35.273193 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:35.273172 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b40367b4-cbb4-4bed-ab22-27752879bbd0-metrics-certs\") pod \"maas-controller-8f8d984c8-dbdv8\" (UID: \"b40367b4-cbb4-4bed-ab22-27752879bbd0\") " pod="opendatahub/maas-controller-8f8d984c8-dbdv8" Sep 11 01:06:35.273193 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:35.273183 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b40367b4-cbb4-4bed-ab22-27752879bbd0-webhook-cert\") pod \"maas-controller-8f8d984c8-dbdv8\" (UID: \"b40367b4-cbb4-4bed-ab22-27752879bbd0\") " pod="opendatahub/maas-controller-8f8d984c8-dbdv8" Sep 11 01:06:35.479631 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:35.479598 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8f8d984c8-dbdv8" Sep 11 01:06:35.752311 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:35.752286 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8f8d984c8-dbdv8"] Sep 11 01:06:35.754903 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:06:35.754878 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb40367b4_cbb4_4bed_ab22_27752879bbd0.slice/crio-dc8145064676f6bebc7620262427e34fa2ab5b6ae3d20274e48a5918b5851ec8 WatchSource:0}: Error finding container dc8145064676f6bebc7620262427e34fa2ab5b6ae3d20274e48a5918b5851ec8: Status 404 returned error can't find the container with id dc8145064676f6bebc7620262427e34fa2ab5b6ae3d20274e48a5918b5851ec8 Sep 11 01:06:35.884101 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:35.884064 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8f8d984c8-dbdv8" event={"ID":"b40367b4-cbb4-4bed-ab22-27752879bbd0","Type":"ContainerStarted","Data":"dc8145064676f6bebc7620262427e34fa2ab5b6ae3d20274e48a5918b5851ec8"} Sep 11 01:06:38.227522 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:38.227483 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:06:38.326907 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:38.326798 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:06:38.900365 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:38.900327 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8f8d984c8-dbdv8" event={"ID":"b40367b4-cbb4-4bed-ab22-27752879bbd0","Type":"ContainerStarted","Data":"b10eef12c723b85220f52c5034b984e139195b6edeaf7566ea82bad9ecd57c26"} Sep 11 01:06:38.900514 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:38.900405 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8f8d984c8-dbdv8" Sep 11 01:06:38.921627 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:38.921574 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-8f8d984c8-dbdv8" podStartSLOduration=2.6349593220000003 podStartE2EDuration="4.921556039s" totalImagesPullingTime="2.286596717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:06:38.918540337 +0000 UTC m=+525.741993850" watchObservedRunningTime="2026-09-11 01:06:38.921556039 +0000 UTC m=+525.745009551" Sep 11 01:06:49.906835 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:06:49.906760 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8f8d984c8-dbdv8" Sep 11 01:07:07.936118 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:07.936081 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:07:08.029569 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:08.029537 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:07:09.358836 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:09.358804 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:07:09.480888 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:09.480836 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:07:09.494920 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:09.494894 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5"] Sep 11 01:07:09.499676 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:09.499653 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5" Sep 11 01:07:09.502033 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:09.502004 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 11 01:07:09.502895 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:09.502876 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fzpvb\"" Sep 11 01:07:09.502990 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:09.502884 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 01:07:09.508257 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:09.508054 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5"] Sep 11 01:07:09.564200 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:09.564165 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:07:09.615616 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:09.615554 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba4726e-2615-4686-9fb9-f490c7886591-openshift-service-ca\") pod \"maas-api-7b7ff8bc87-b6ls5\" (UID: \"4ba4726e-2615-4686-9fb9-f490c7886591\") " pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5" Sep 11 01:07:09.615616 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:09.615586 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vb68\" (UniqueName: \"kubernetes.io/projected/4ba4726e-2615-4686-9fb9-f490c7886591-kube-api-access-6vb68\") pod \"maas-api-7b7ff8bc87-b6ls5\" (UID: \"4ba4726e-2615-4686-9fb9-f490c7886591\") " pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5" Sep 11 01:07:09.615616 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:09.615607 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ba4726e-2615-4686-9fb9-f490c7886591-maas-api-tls\") pod \"maas-api-7b7ff8bc87-b6ls5\" (UID: \"4ba4726e-2615-4686-9fb9-f490c7886591\") " pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5" Sep 11 01:07:09.615783 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:09.615675 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ba4726e-2615-4686-9fb9-f490c7886591-metrics-certs\") pod \"maas-api-7b7ff8bc87-b6ls5\" (UID: \"4ba4726e-2615-4686-9fb9-f490c7886591\") " pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5" Sep 11 01:07:09.662927 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:09.662903 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:07:09.716589 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:09.716562 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba4726e-2615-4686-9fb9-f490c7886591-openshift-service-ca\") pod \"maas-api-7b7ff8bc87-b6ls5\" (UID: \"4ba4726e-2615-4686-9fb9-f490c7886591\") " pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5" Sep 11 01:07:09.716687 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:09.716592 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vb68\" (UniqueName: \"kubernetes.io/projected/4ba4726e-2615-4686-9fb9-f490c7886591-kube-api-access-6vb68\") pod \"maas-api-7b7ff8bc87-b6ls5\" (UID: \"4ba4726e-2615-4686-9fb9-f490c7886591\") " pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5" Sep 11 01:07:09.716687 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:09.716612 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ba4726e-2615-4686-9fb9-f490c7886591-maas-api-tls\") pod \"maas-api-7b7ff8bc87-b6ls5\" (UID: \"4ba4726e-2615-4686-9fb9-f490c7886591\") " pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5" Sep 11 01:07:09.716823 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:09.716807 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ba4726e-2615-4686-9fb9-f490c7886591-metrics-certs\") pod \"maas-api-7b7ff8bc87-b6ls5\" (UID: \"4ba4726e-2615-4686-9fb9-f490c7886591\") " pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5" Sep 11 01:07:09.717307 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:09.717285 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba4726e-2615-4686-9fb9-f490c7886591-openshift-service-ca\") pod \"maas-api-7b7ff8bc87-b6ls5\" (UID: \"4ba4726e-2615-4686-9fb9-f490c7886591\") " pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5" Sep 11 01:07:09.719239 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:09.719220 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ba4726e-2615-4686-9fb9-f490c7886591-metrics-certs\") pod \"maas-api-7b7ff8bc87-b6ls5\" (UID: \"4ba4726e-2615-4686-9fb9-f490c7886591\") " pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5" Sep 11 01:07:09.719295 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:09.719239 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ba4726e-2615-4686-9fb9-f490c7886591-maas-api-tls\") pod \"maas-api-7b7ff8bc87-b6ls5\" (UID: \"4ba4726e-2615-4686-9fb9-f490c7886591\") " pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5" Sep 11 01:07:09.725061 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:09.725034 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vb68\" (UniqueName: \"kubernetes.io/projected/4ba4726e-2615-4686-9fb9-f490c7886591-kube-api-access-6vb68\") pod \"maas-api-7b7ff8bc87-b6ls5\" (UID: \"4ba4726e-2615-4686-9fb9-f490c7886591\") " pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5" Sep 11 01:07:09.812896 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:09.812866 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5" Sep 11 01:07:09.931886 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:09.931794 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:07:10.029089 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:10.029057 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:07:10.090139 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:10.090118 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5"] Sep 11 01:07:10.092228 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:07:10.092201 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba4726e_2615_4686_9fb9_f490c7886591.slice/crio-fa9c884d1331af30a1272d1b03f45791d241f0e1298df29614a9e595442443b8 WatchSource:0}: Error finding container fa9c884d1331af30a1272d1b03f45791d241f0e1298df29614a9e595442443b8: Status 404 returned error can't find the container with id fa9c884d1331af30a1272d1b03f45791d241f0e1298df29614a9e595442443b8 Sep 11 01:07:11.048482 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:11.048432 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5" event={"ID":"4ba4726e-2615-4686-9fb9-f490c7886591","Type":"ContainerStarted","Data":"fa9c884d1331af30a1272d1b03f45791d241f0e1298df29614a9e595442443b8"} Sep 11 01:07:13.059934 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:13.059896 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5" event={"ID":"4ba4726e-2615-4686-9fb9-f490c7886591","Type":"ContainerStarted","Data":"b5e39a40e5ecc16bc2404eca491216ca9046a2785edb9a3196231a1df1e1112c"} Sep 11 01:07:13.060305 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:13.060037 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5" Sep 11 01:07:13.076968 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:13.076919 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5" podStartSLOduration=2.188052892 podStartE2EDuration="4.07690339s" totalImagesPullingTime="1.888850498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:07:13.075326939 +0000 UTC m=+559.898780450" watchObservedRunningTime="2026-09-11 01:07:13.07690339 +0000 UTC m=+559.900356900" Sep 11 01:07:19.070599 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:19.070568 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5" Sep 11 01:07:20.189325 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:20.189288 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:07:20.297576 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:20.297543 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:07:24.902417 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:24.902379 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq"] Sep 11 01:07:24.904977 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:24.904948 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" Sep 11 01:07:24.907410 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:24.907382 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 11 01:07:24.907410 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:24.907383 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 11 01:07:24.908227 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:24.908207 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 11 01:07:24.908310 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:24.908260 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lg79n\"" Sep 11 01:07:24.914532 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:24.914507 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq"] Sep 11 01:07:24.921037 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:24.921010 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:07:25.034860 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:25.034812 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/518aded8-c7a6-4bf7-adf7-35117bb089c9-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq\" (UID: \"518aded8-c7a6-4bf7-adf7-35117bb089c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" Sep 11 01:07:25.034860 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:25.034861 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/518aded8-c7a6-4bf7-adf7-35117bb089c9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq\" (UID: \"518aded8-c7a6-4bf7-adf7-35117bb089c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" Sep 11 01:07:25.035031 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:25.034885 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htgpm\" (UniqueName: \"kubernetes.io/projected/518aded8-c7a6-4bf7-adf7-35117bb089c9-kube-api-access-htgpm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq\" (UID: \"518aded8-c7a6-4bf7-adf7-35117bb089c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" Sep 11 01:07:25.035031 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:25.034967 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/518aded8-c7a6-4bf7-adf7-35117bb089c9-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq\" (UID: \"518aded8-c7a6-4bf7-adf7-35117bb089c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" Sep 11 01:07:25.035031 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:25.034996 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/518aded8-c7a6-4bf7-adf7-35117bb089c9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq\" (UID: \"518aded8-c7a6-4bf7-adf7-35117bb089c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" Sep 11 01:07:25.035031 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:25.035027 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/518aded8-c7a6-4bf7-adf7-35117bb089c9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq\" (UID: \"518aded8-c7a6-4bf7-adf7-35117bb089c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" Sep 11 01:07:25.135742 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:25.135717 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/518aded8-c7a6-4bf7-adf7-35117bb089c9-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq\" (UID: \"518aded8-c7a6-4bf7-adf7-35117bb089c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" Sep 11 01:07:25.135881 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:25.135748 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/518aded8-c7a6-4bf7-adf7-35117bb089c9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq\" (UID: \"518aded8-c7a6-4bf7-adf7-35117bb089c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" Sep 11 01:07:25.135881 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:25.135770 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htgpm\" (UniqueName: \"kubernetes.io/projected/518aded8-c7a6-4bf7-adf7-35117bb089c9-kube-api-access-htgpm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq\" (UID: \"518aded8-c7a6-4bf7-adf7-35117bb089c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" Sep 11 01:07:25.135881 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:25.135793 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/518aded8-c7a6-4bf7-adf7-35117bb089c9-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq\" (UID: \"518aded8-c7a6-4bf7-adf7-35117bb089c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" Sep 11 01:07:25.135881 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:25.135809 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/518aded8-c7a6-4bf7-adf7-35117bb089c9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq\" (UID: \"518aded8-c7a6-4bf7-adf7-35117bb089c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" Sep 11 01:07:25.135881 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:25.135834 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/518aded8-c7a6-4bf7-adf7-35117bb089c9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq\" (UID: \"518aded8-c7a6-4bf7-adf7-35117bb089c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" Sep 11 01:07:25.136220 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:25.136191 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/518aded8-c7a6-4bf7-adf7-35117bb089c9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq\" (UID: \"518aded8-c7a6-4bf7-adf7-35117bb089c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" Sep 11 01:07:25.136324 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:25.136236 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/518aded8-c7a6-4bf7-adf7-35117bb089c9-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq\" (UID: \"518aded8-c7a6-4bf7-adf7-35117bb089c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" Sep 11 01:07:25.136324 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:25.136282 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/518aded8-c7a6-4bf7-adf7-35117bb089c9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq\" (UID: \"518aded8-c7a6-4bf7-adf7-35117bb089c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" Sep 11 01:07:25.137987 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:25.137960 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/518aded8-c7a6-4bf7-adf7-35117bb089c9-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq\" (UID: \"518aded8-c7a6-4bf7-adf7-35117bb089c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" Sep 11 01:07:25.138232 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:25.138217 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/518aded8-c7a6-4bf7-adf7-35117bb089c9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq\" (UID: \"518aded8-c7a6-4bf7-adf7-35117bb089c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" Sep 11 01:07:25.143281 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:25.143250 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htgpm\" (UniqueName: \"kubernetes.io/projected/518aded8-c7a6-4bf7-adf7-35117bb089c9-kube-api-access-htgpm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq\" (UID: \"518aded8-c7a6-4bf7-adf7-35117bb089c9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" Sep 11 01:07:25.215935 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:25.215905 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" Sep 11 01:07:25.492824 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:25.492798 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq"] Sep 11 01:07:25.495085 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:07:25.495056 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod518aded8_c7a6_4bf7_adf7_35117bb089c9.slice/crio-42015141ca2192001e73abee80e6f601171c4412990a00957be68bd5012057b4 WatchSource:0}: Error finding container 42015141ca2192001e73abee80e6f601171c4412990a00957be68bd5012057b4: Status 404 returned error can't find the container with id 42015141ca2192001e73abee80e6f601171c4412990a00957be68bd5012057b4 Sep 11 01:07:26.122172 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:26.122136 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" event={"ID":"518aded8-c7a6-4bf7-adf7-35117bb089c9","Type":"ContainerStarted","Data":"42015141ca2192001e73abee80e6f601171c4412990a00957be68bd5012057b4"} Sep 11 01:07:27.128632 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:27.128589 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" event={"ID":"518aded8-c7a6-4bf7-adf7-35117bb089c9","Type":"ContainerStarted","Data":"5dc0cad7ff83dac8937fb21ddbccec9a3fda90ce8685925d8d571667f436d2df"} Sep 11 01:07:27.145757 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:27.145710 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" podStartSLOduration=2.067131782 podStartE2EDuration="3.145694066s" totalImagesPullingTime="1.078562284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:07:27.144927802 +0000 UTC m=+573.968381317" watchObservedRunningTime="2026-09-11 01:07:27.145694066 +0000 UTC m=+573.969147575" Sep 11 01:07:30.977652 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:30.977604 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:07:31.759599 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:31.759557 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:07:31.813640 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:31.813610 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:07:35.216831 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:35.216797 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" Sep 11 01:07:35.217192 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:35.216878 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" Sep 11 01:07:35.229555 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:35.229535 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" Sep 11 01:07:36.182067 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:36.182036 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq" Sep 11 01:07:38.011477 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:38.011443 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:07:44.623417 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:44.623378 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:07:53.604271 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:53.604242 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wtd8v_c2b320dd-43e9-435d-8e4f-2bca1f1233ff/ovn-acl-logging/0.log" Sep 11 01:07:53.604674 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:53.604280 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wtd8v_c2b320dd-43e9-435d-8e4f-2bca1f1233ff/ovn-acl-logging/0.log" Sep 11 01:07:56.709231 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:56.709194 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr"] Sep 11 01:07:56.711373 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:56.711357 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" Sep 11 01:07:56.713449 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:56.713426 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 11 01:07:56.721798 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:56.721778 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr"] Sep 11 01:07:56.785308 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:56.785278 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:07:56.890352 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:56.890325 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f5a6809-9c04-48b5-9efe-0104d3940ae6-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-84kdr\" (UID: \"1f5a6809-9c04-48b5-9efe-0104d3940ae6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" Sep 11 01:07:56.890513 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:56.890356 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f5a6809-9c04-48b5-9efe-0104d3940ae6-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-84kdr\" (UID: \"1f5a6809-9c04-48b5-9efe-0104d3940ae6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" Sep 11 01:07:56.890513 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:56.890375 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnl4c\" (UniqueName: \"kubernetes.io/projected/1f5a6809-9c04-48b5-9efe-0104d3940ae6-kube-api-access-pnl4c\") pod \"e2e-trlp-test-simulated-kserve-598466548d-84kdr\" (UID: \"1f5a6809-9c04-48b5-9efe-0104d3940ae6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" Sep 11 01:07:56.890513 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:56.890400 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f5a6809-9c04-48b5-9efe-0104d3940ae6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-84kdr\" (UID: \"1f5a6809-9c04-48b5-9efe-0104d3940ae6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" Sep 11 01:07:56.890513 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:56.890503 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f5a6809-9c04-48b5-9efe-0104d3940ae6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-84kdr\" (UID: \"1f5a6809-9c04-48b5-9efe-0104d3940ae6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" Sep 11 01:07:56.890717 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:56.890576 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f5a6809-9c04-48b5-9efe-0104d3940ae6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-84kdr\" (UID: \"1f5a6809-9c04-48b5-9efe-0104d3940ae6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" Sep 11 01:07:56.991324 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:56.991260 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f5a6809-9c04-48b5-9efe-0104d3940ae6-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-84kdr\" (UID: \"1f5a6809-9c04-48b5-9efe-0104d3940ae6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" Sep 11 01:07:56.991324 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:56.991293 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f5a6809-9c04-48b5-9efe-0104d3940ae6-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-84kdr\" (UID: \"1f5a6809-9c04-48b5-9efe-0104d3940ae6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" Sep 11 01:07:56.991324 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:56.991312 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnl4c\" (UniqueName: \"kubernetes.io/projected/1f5a6809-9c04-48b5-9efe-0104d3940ae6-kube-api-access-pnl4c\") pod \"e2e-trlp-test-simulated-kserve-598466548d-84kdr\" (UID: \"1f5a6809-9c04-48b5-9efe-0104d3940ae6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" Sep 11 01:07:56.991541 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:56.991338 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f5a6809-9c04-48b5-9efe-0104d3940ae6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-84kdr\" (UID: \"1f5a6809-9c04-48b5-9efe-0104d3940ae6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" Sep 11 01:07:56.991541 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:56.991382 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f5a6809-9c04-48b5-9efe-0104d3940ae6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-84kdr\" (UID: \"1f5a6809-9c04-48b5-9efe-0104d3940ae6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" Sep 11 01:07:56.991541 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:56.991433 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f5a6809-9c04-48b5-9efe-0104d3940ae6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-84kdr\" (UID: \"1f5a6809-9c04-48b5-9efe-0104d3940ae6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" Sep 11 01:07:56.991698 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:56.991684 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f5a6809-9c04-48b5-9efe-0104d3940ae6-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-84kdr\" (UID: \"1f5a6809-9c04-48b5-9efe-0104d3940ae6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" Sep 11 01:07:56.991814 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:56.991797 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f5a6809-9c04-48b5-9efe-0104d3940ae6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-84kdr\" (UID: \"1f5a6809-9c04-48b5-9efe-0104d3940ae6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" Sep 11 01:07:56.991926 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:56.991906 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f5a6809-9c04-48b5-9efe-0104d3940ae6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-84kdr\" (UID: \"1f5a6809-9c04-48b5-9efe-0104d3940ae6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" Sep 11 01:07:56.993801 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:56.993771 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f5a6809-9c04-48b5-9efe-0104d3940ae6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-84kdr\" (UID: \"1f5a6809-9c04-48b5-9efe-0104d3940ae6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" Sep 11 01:07:56.993917 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:56.993892 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f5a6809-9c04-48b5-9efe-0104d3940ae6-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-84kdr\" (UID: \"1f5a6809-9c04-48b5-9efe-0104d3940ae6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" Sep 11 01:07:56.998769 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:56.998748 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnl4c\" (UniqueName: \"kubernetes.io/projected/1f5a6809-9c04-48b5-9efe-0104d3940ae6-kube-api-access-pnl4c\") pod \"e2e-trlp-test-simulated-kserve-598466548d-84kdr\" (UID: \"1f5a6809-9c04-48b5-9efe-0104d3940ae6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" Sep 11 01:07:57.021597 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:57.021574 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" Sep 11 01:07:57.287356 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:57.287326 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr"] Sep 11 01:07:57.289397 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:07:57.289373 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f5a6809_9c04_48b5_9efe_0104d3940ae6.slice/crio-41724d43f4519eb50f642a381b3d3f7ea44c48beedf1e4f663b2e653819bcbe2 WatchSource:0}: Error finding container 41724d43f4519eb50f642a381b3d3f7ea44c48beedf1e4f663b2e653819bcbe2: Status 404 returned error can't find the container with id 41724d43f4519eb50f642a381b3d3f7ea44c48beedf1e4f663b2e653819bcbe2 Sep 11 01:07:58.267382 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:58.267341 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" event={"ID":"1f5a6809-9c04-48b5-9efe-0104d3940ae6","Type":"ContainerStarted","Data":"ebf0679074c47b5a33c8c151239904dd15d2fee256e99bc8cc5981241c5f05c5"} Sep 11 01:07:58.267743 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:58.267400 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" event={"ID":"1f5a6809-9c04-48b5-9efe-0104d3940ae6","Type":"ContainerStarted","Data":"41724d43f4519eb50f642a381b3d3f7ea44c48beedf1e4f663b2e653819bcbe2"} Sep 11 01:07:58.284149 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:07:58.284102 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" podStartSLOduration=2.104793898 podStartE2EDuration="2.284086485s" totalImagesPullingTime="179.292587ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:07:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:07:58.282955621 +0000 UTC m=+605.106409131" watchObservedRunningTime="2026-09-11 01:07:58.284086485 +0000 UTC m=+605.107539999" Sep 11 01:08:07.022526 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:07.022492 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" Sep 11 01:08:07.022909 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:07.022540 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" Sep 11 01:08:07.034539 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:07.034514 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" Sep 11 01:08:07.321240 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:07.321171 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-84kdr" Sep 11 01:08:15.289553 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.289517 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc"] Sep 11 01:08:15.294214 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.294191 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" Sep 11 01:08:15.296498 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.296474 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 11 01:08:15.302546 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.302519 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc"] Sep 11 01:08:15.313522 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.313491 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9982d7-eb58-4c79-af91-56bc80c6c21b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc\" (UID: \"bb9982d7-eb58-4c79-af91-56bc80c6c21b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" Sep 11 01:08:15.313629 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.313527 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9982d7-eb58-4c79-af91-56bc80c6c21b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc\" (UID: \"bb9982d7-eb58-4c79-af91-56bc80c6c21b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" Sep 11 01:08:15.313629 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.313560 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9982d7-eb58-4c79-af91-56bc80c6c21b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc\" (UID: \"bb9982d7-eb58-4c79-af91-56bc80c6c21b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" Sep 11 01:08:15.313629 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.313619 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb9982d7-eb58-4c79-af91-56bc80c6c21b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc\" (UID: \"bb9982d7-eb58-4c79-af91-56bc80c6c21b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" Sep 11 01:08:15.313788 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.313654 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjk74\" (UniqueName: \"kubernetes.io/projected/bb9982d7-eb58-4c79-af91-56bc80c6c21b-kube-api-access-tjk74\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc\" (UID: \"bb9982d7-eb58-4c79-af91-56bc80c6c21b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" Sep 11 01:08:15.313788 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.313749 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9982d7-eb58-4c79-af91-56bc80c6c21b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc\" (UID: \"bb9982d7-eb58-4c79-af91-56bc80c6c21b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" Sep 11 01:08:15.326122 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.326099 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:08:15.414397 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.414368 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjk74\" (UniqueName: \"kubernetes.io/projected/bb9982d7-eb58-4c79-af91-56bc80c6c21b-kube-api-access-tjk74\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc\" (UID: \"bb9982d7-eb58-4c79-af91-56bc80c6c21b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" Sep 11 01:08:15.414578 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.414420 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9982d7-eb58-4c79-af91-56bc80c6c21b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc\" (UID: \"bb9982d7-eb58-4c79-af91-56bc80c6c21b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" Sep 11 01:08:15.414578 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.414479 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9982d7-eb58-4c79-af91-56bc80c6c21b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc\" (UID: \"bb9982d7-eb58-4c79-af91-56bc80c6c21b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" Sep 11 01:08:15.414578 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.414505 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9982d7-eb58-4c79-af91-56bc80c6c21b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc\" (UID: \"bb9982d7-eb58-4c79-af91-56bc80c6c21b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" Sep 11 01:08:15.414578 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.414540 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9982d7-eb58-4c79-af91-56bc80c6c21b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc\" (UID: \"bb9982d7-eb58-4c79-af91-56bc80c6c21b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" Sep 11 01:08:15.414900 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.414604 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb9982d7-eb58-4c79-af91-56bc80c6c21b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc\" (UID: \"bb9982d7-eb58-4c79-af91-56bc80c6c21b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" Sep 11 01:08:15.414900 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.414831 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb9982d7-eb58-4c79-af91-56bc80c6c21b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc\" (UID: \"bb9982d7-eb58-4c79-af91-56bc80c6c21b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" Sep 11 01:08:15.415075 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.415045 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb9982d7-eb58-4c79-af91-56bc80c6c21b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc\" (UID: \"bb9982d7-eb58-4c79-af91-56bc80c6c21b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" Sep 11 01:08:15.415178 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.415096 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb9982d7-eb58-4c79-af91-56bc80c6c21b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc\" (UID: \"bb9982d7-eb58-4c79-af91-56bc80c6c21b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" Sep 11 01:08:15.416790 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.416770 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb9982d7-eb58-4c79-af91-56bc80c6c21b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc\" (UID: \"bb9982d7-eb58-4c79-af91-56bc80c6c21b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" Sep 11 01:08:15.417458 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.417439 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb9982d7-eb58-4c79-af91-56bc80c6c21b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc\" (UID: \"bb9982d7-eb58-4c79-af91-56bc80c6c21b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" Sep 11 01:08:15.422809 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.422787 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjk74\" (UniqueName: \"kubernetes.io/projected/bb9982d7-eb58-4c79-af91-56bc80c6c21b-kube-api-access-tjk74\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc\" (UID: \"bb9982d7-eb58-4c79-af91-56bc80c6c21b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" Sep 11 01:08:15.607109 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.607050 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" Sep 11 01:08:15.875639 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.875613 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc"] Sep 11 01:08:15.877980 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:08:15.877948 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb9982d7_eb58_4c79_af91_56bc80c6c21b.slice/crio-850902fac6de773172012fd808afcfe50d6c25d727639470b93873109d5f48d0 WatchSource:0}: Error finding container 850902fac6de773172012fd808afcfe50d6c25d727639470b93873109d5f48d0: Status 404 returned error can't find the container with id 850902fac6de773172012fd808afcfe50d6c25d727639470b93873109d5f48d0 Sep 11 01:08:15.879345 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:15.879327 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 01:08:16.353384 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:16.353348 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" event={"ID":"bb9982d7-eb58-4c79-af91-56bc80c6c21b","Type":"ContainerStarted","Data":"e84babca97ea8802d135a8fdc56bbbbb9017510fa6754b9b823290165b24c619"} Sep 11 01:08:16.353748 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:16.353406 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" event={"ID":"bb9982d7-eb58-4c79-af91-56bc80c6c21b","Type":"ContainerStarted","Data":"850902fac6de773172012fd808afcfe50d6c25d727639470b93873109d5f48d0"} Sep 11 01:08:16.370451 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:16.370406 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" podStartSLOduration=1.183483204 podStartE2EDuration="1.370390986s" totalImagesPullingTime="186.907782ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:08:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:08:16.368605964 +0000 UTC m=+623.192059476" watchObservedRunningTime="2026-09-11 01:08:16.370390986 +0000 UTC m=+623.193844495" Sep 11 01:08:25.607913 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:25.607878 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" Sep 11 01:08:25.608363 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:25.607939 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" Sep 11 01:08:25.620245 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:25.620218 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" Sep 11 01:08:26.407660 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:26.407631 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc" Sep 11 01:08:35.598022 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:35.597982 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:08:49.330483 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:08:49.330449 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:09:17.203624 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:17.203585 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:09:34.892116 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:34.892078 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:09:34.957142 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:34.957116 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:09:37.021943 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:37.021907 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-78cd557cc4-ptpp5"] Sep 11 01:09:37.025491 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:37.025468 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78cd557cc4-ptpp5" Sep 11 01:09:37.025835 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:37.025814 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl2j7\" (UniqueName: \"kubernetes.io/projected/39c32d6f-b4ab-4f08-b9d8-2b5fd0998ba5-kube-api-access-kl2j7\") pod \"authorino-78cd557cc4-ptpp5\" (UID: \"39c32d6f-b4ab-4f08-b9d8-2b5fd0998ba5\") " pod="kuadrant-system/authorino-78cd557cc4-ptpp5" Sep 11 01:09:37.025937 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:37.025925 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/39c32d6f-b4ab-4f08-b9d8-2b5fd0998ba5-tls-cert\") pod \"authorino-78cd557cc4-ptpp5\" (UID: \"39c32d6f-b4ab-4f08-b9d8-2b5fd0998ba5\") " pod="kuadrant-system/authorino-78cd557cc4-ptpp5" Sep 11 01:09:37.028929 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:37.028911 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 11 01:09:37.029028 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:37.028915 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bjhvj\"" Sep 11 01:09:37.038770 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:37.038746 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78cd557cc4-ptpp5"] Sep 11 01:09:37.126512 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:37.126473 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/39c32d6f-b4ab-4f08-b9d8-2b5fd0998ba5-tls-cert\") pod \"authorino-78cd557cc4-ptpp5\" (UID: \"39c32d6f-b4ab-4f08-b9d8-2b5fd0998ba5\") " pod="kuadrant-system/authorino-78cd557cc4-ptpp5" Sep 11 01:09:37.126677 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:37.126527 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl2j7\" (UniqueName: \"kubernetes.io/projected/39c32d6f-b4ab-4f08-b9d8-2b5fd0998ba5-kube-api-access-kl2j7\") pod \"authorino-78cd557cc4-ptpp5\" (UID: \"39c32d6f-b4ab-4f08-b9d8-2b5fd0998ba5\") " pod="kuadrant-system/authorino-78cd557cc4-ptpp5" Sep 11 01:09:37.128879 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:37.128831 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/39c32d6f-b4ab-4f08-b9d8-2b5fd0998ba5-tls-cert\") pod \"authorino-78cd557cc4-ptpp5\" (UID: \"39c32d6f-b4ab-4f08-b9d8-2b5fd0998ba5\") " pod="kuadrant-system/authorino-78cd557cc4-ptpp5" Sep 11 01:09:37.134622 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:37.134599 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl2j7\" (UniqueName: \"kubernetes.io/projected/39c32d6f-b4ab-4f08-b9d8-2b5fd0998ba5-kube-api-access-kl2j7\") pod \"authorino-78cd557cc4-ptpp5\" (UID: \"39c32d6f-b4ab-4f08-b9d8-2b5fd0998ba5\") " pod="kuadrant-system/authorino-78cd557cc4-ptpp5" Sep 11 01:09:37.334895 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:37.334792 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78cd557cc4-ptpp5" Sep 11 01:09:37.604303 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:37.604232 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78cd557cc4-ptpp5"] Sep 11 01:09:37.608514 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:09:37.608488 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39c32d6f_b4ab_4f08_b9d8_2b5fd0998ba5.slice/crio-b35983e347e2f50ebaa88bfdf9e5fb5e45efb6eec91bd854ccc2d0ca7949ff2f WatchSource:0}: Error finding container b35983e347e2f50ebaa88bfdf9e5fb5e45efb6eec91bd854ccc2d0ca7949ff2f: Status 404 returned error can't find the container with id b35983e347e2f50ebaa88bfdf9e5fb5e45efb6eec91bd854ccc2d0ca7949ff2f Sep 11 01:09:37.727721 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:37.727684 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78cd557cc4-ptpp5" event={"ID":"39c32d6f-b4ab-4f08-b9d8-2b5fd0998ba5","Type":"ContainerStarted","Data":"b35983e347e2f50ebaa88bfdf9e5fb5e45efb6eec91bd854ccc2d0ca7949ff2f"} Sep 11 01:09:38.735238 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:38.735191 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78cd557cc4-ptpp5" event={"ID":"39c32d6f-b4ab-4f08-b9d8-2b5fd0998ba5","Type":"ContainerStarted","Data":"649e536cdd07a75e1392a02eee8d002c769aa347be2839d0b3e0f0dc9782287e"} Sep 11 01:09:38.758346 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:38.758267 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-78cd557cc4-ptpp5" podStartSLOduration=1.12358759 podStartE2EDuration="1.758247008s" totalImagesPullingTime="634.659418ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:09:38.755987495 +0000 UTC m=+705.579441006" watchObservedRunningTime="2026-09-11 01:09:38.758247008 +0000 UTC m=+705.581700522" Sep 11 01:09:39.461200 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:39.461161 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:09:39.622538 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:39.622479 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:09:40.774825 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:40.774785 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:09:40.925040 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:40.925000 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:09:56.687537 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:56.687449 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:09:56.746189 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:56.746160 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:09:56.812483 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:56.812453 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:09:57.021972 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:57.021887 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:09:57.125229 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:57.125199 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:09:57.219470 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:57.219427 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:09:57.319281 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:57.319200 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:09:57.381732 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:57.381701 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:09:57.503185 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:57.503154 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:09:58.158958 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:58.158920 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:09:58.249674 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:58.249635 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:09:58.605189 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:09:58.605158 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:00.108696 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:00.108662 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:00.223040 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:00.223003 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:00.317149 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:00.317117 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:00.413595 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:00.413515 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:00.513287 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:00.513249 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:01.409832 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:01.409797 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk"] Sep 11 01:10:01.413835 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:01.413813 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" Sep 11 01:10:01.416275 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:01.416248 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-651569-serving-cert\"" Sep 11 01:10:01.426636 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:01.426609 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk"] Sep 11 01:10:01.469906 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:01.469875 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:01.514217 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:01.514189 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxc6p\" (UniqueName: \"kubernetes.io/projected/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-kube-api-access-hxc6p\") pod \"maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk\" (UID: \"07f53eed-c8ba-42ee-ac68-f1d1cd2594db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" Sep 11 01:10:01.514346 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:01.514235 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk\" (UID: \"07f53eed-c8ba-42ee-ac68-f1d1cd2594db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" Sep 11 01:10:01.514346 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:01.514308 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-maas-api-tls\") pod \"maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk\" (UID: \"07f53eed-c8ba-42ee-ac68-f1d1cd2594db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" Sep 11 01:10:01.514346 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:01.514328 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-metrics-certs\") pod \"maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk\" (UID: \"07f53eed-c8ba-42ee-ac68-f1d1cd2594db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" Sep 11 01:10:01.587585 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:01.587553 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:01.615431 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:01.615400 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxc6p\" (UniqueName: \"kubernetes.io/projected/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-kube-api-access-hxc6p\") pod \"maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk\" (UID: \"07f53eed-c8ba-42ee-ac68-f1d1cd2594db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" Sep 11 01:10:01.615797 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:01.615465 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk\" (UID: \"07f53eed-c8ba-42ee-ac68-f1d1cd2594db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" Sep 11 01:10:01.615797 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:01.615536 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-maas-api-tls\") pod \"maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk\" (UID: \"07f53eed-c8ba-42ee-ac68-f1d1cd2594db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" Sep 11 01:10:01.615797 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:01.615559 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-metrics-certs\") pod \"maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk\" (UID: \"07f53eed-c8ba-42ee-ac68-f1d1cd2594db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" Sep 11 01:10:01.615797 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:10:01.615703 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-serving-cert: secret "maas-api-e2e-worker-w4-651569-serving-cert" not found Sep 11 01:10:01.615797 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:10:01.615784 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-maas-api-tls podName:07f53eed-c8ba-42ee-ac68-f1d1cd2594db nodeName:}" failed. No retries permitted until 2026-09-11 01:10:02.115762009 +0000 UTC m=+728.939215502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-maas-api-tls") pod "maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" (UID: "07f53eed-c8ba-42ee-ac68-f1d1cd2594db") : secret "maas-api-e2e-worker-w4-651569-serving-cert" not found Sep 11 01:10:01.616159 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:01.616135 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk\" (UID: \"07f53eed-c8ba-42ee-ac68-f1d1cd2594db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" Sep 11 01:10:01.618174 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:01.618145 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-metrics-certs\") pod \"maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk\" (UID: \"07f53eed-c8ba-42ee-ac68-f1d1cd2594db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" Sep 11 01:10:01.628885 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:01.628860 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxc6p\" (UniqueName: \"kubernetes.io/projected/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-kube-api-access-hxc6p\") pod \"maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk\" (UID: \"07f53eed-c8ba-42ee-ac68-f1d1cd2594db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" Sep 11 01:10:02.092873 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:02.092822 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:02.121224 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:02.121195 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-maas-api-tls\") pod \"maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk\" (UID: \"07f53eed-c8ba-42ee-ac68-f1d1cd2594db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" Sep 11 01:10:02.121379 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:10:02.121334 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-serving-cert: secret "maas-api-e2e-worker-w4-651569-serving-cert" not found Sep 11 01:10:02.121441 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:10:02.121392 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-maas-api-tls podName:07f53eed-c8ba-42ee-ac68-f1d1cd2594db nodeName:}" failed. No retries permitted until 2026-09-11 01:10:03.121378264 +0000 UTC m=+729.944831752 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-maas-api-tls") pod "maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" (UID: "07f53eed-c8ba-42ee-ac68-f1d1cd2594db") : secret "maas-api-e2e-worker-w4-651569-serving-cert" not found Sep 11 01:10:02.186507 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:02.186471 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:02.358508 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:02.358417 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:02.494067 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:02.494031 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg"] Sep 11 01:10:02.497701 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:02.497677 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg" Sep 11 01:10:02.500099 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:02.500076 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-cef92e-serving-cert\"" Sep 11 01:10:02.507470 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:02.507447 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg"] Sep 11 01:10:02.524491 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:02.524460 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:02.607565 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:02.607533 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:02.625979 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:02.625919 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15915139-3495-41f1-b135-86fd42bf424e-maas-api-tls\") pod \"maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg\" (UID: \"15915139-3495-41f1-b135-86fd42bf424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg" Sep 11 01:10:02.626059 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:02.625997 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15915139-3495-41f1-b135-86fd42bf424e-metrics-certs\") pod \"maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg\" (UID: \"15915139-3495-41f1-b135-86fd42bf424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg" Sep 11 01:10:02.626059 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:02.626044 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15915139-3495-41f1-b135-86fd42bf424e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg\" (UID: \"15915139-3495-41f1-b135-86fd42bf424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg" Sep 11 01:10:02.626140 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:02.626089 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwq2s\" (UniqueName: \"kubernetes.io/projected/15915139-3495-41f1-b135-86fd42bf424e-kube-api-access-rwq2s\") pod \"maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg\" (UID: \"15915139-3495-41f1-b135-86fd42bf424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg" Sep 11 01:10:02.705330 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:02.705300 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:02.727427 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:02.727401 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15915139-3495-41f1-b135-86fd42bf424e-metrics-certs\") pod \"maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg\" (UID: \"15915139-3495-41f1-b135-86fd42bf424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg" Sep 11 01:10:02.727544 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:02.727467 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15915139-3495-41f1-b135-86fd42bf424e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg\" (UID: \"15915139-3495-41f1-b135-86fd42bf424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg" Sep 11 01:10:02.727544 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:02.727508 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwq2s\" (UniqueName: \"kubernetes.io/projected/15915139-3495-41f1-b135-86fd42bf424e-kube-api-access-rwq2s\") pod \"maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg\" (UID: \"15915139-3495-41f1-b135-86fd42bf424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg" Sep 11 01:10:02.727674 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:02.727565 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15915139-3495-41f1-b135-86fd42bf424e-maas-api-tls\") pod \"maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg\" (UID: \"15915139-3495-41f1-b135-86fd42bf424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg" Sep 11 01:10:02.728243 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:02.728219 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15915139-3495-41f1-b135-86fd42bf424e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg\" (UID: \"15915139-3495-41f1-b135-86fd42bf424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg" Sep 11 01:10:02.729909 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:02.729884 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15915139-3495-41f1-b135-86fd42bf424e-maas-api-tls\") pod \"maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg\" (UID: \"15915139-3495-41f1-b135-86fd42bf424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg" Sep 11 01:10:02.729987 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:02.729948 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15915139-3495-41f1-b135-86fd42bf424e-metrics-certs\") pod \"maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg\" (UID: \"15915139-3495-41f1-b135-86fd42bf424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg" Sep 11 01:10:02.736521 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:02.736498 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwq2s\" (UniqueName: \"kubernetes.io/projected/15915139-3495-41f1-b135-86fd42bf424e-kube-api-access-rwq2s\") pod \"maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg\" (UID: \"15915139-3495-41f1-b135-86fd42bf424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg" Sep 11 01:10:02.811258 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:02.811233 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg" Sep 11 01:10:03.009112 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.009008 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:03.083833 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.083811 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg"] Sep 11 01:10:03.085785 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:10:03.085760 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15915139_3495_41f1_b135_86fd42bf424e.slice/crio-0c66cfe5bf4203bf8b85d4f9a76fec8cf29c9ebfa827dcc3f9e938fe16114b69 WatchSource:0}: Error finding container 0c66cfe5bf4203bf8b85d4f9a76fec8cf29c9ebfa827dcc3f9e938fe16114b69: Status 404 returned error can't find the container with id 0c66cfe5bf4203bf8b85d4f9a76fec8cf29c9ebfa827dcc3f9e938fe16114b69 Sep 11 01:10:03.111656 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.111623 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:03.131189 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.131157 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-maas-api-tls\") pod \"maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk\" (UID: \"07f53eed-c8ba-42ee-ac68-f1d1cd2594db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" Sep 11 01:10:03.133465 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.133440 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-maas-api-tls\") pod \"maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk\" (UID: \"07f53eed-c8ba-42ee-ac68-f1d1cd2594db\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" Sep 11 01:10:03.227689 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.227667 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" Sep 11 01:10:03.526871 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.526824 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk"] Sep 11 01:10:03.532175 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:10:03.532140 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f53eed_c8ba_42ee_ac68_f1d1cd2594db.slice/crio-543a4a6407b17214c7c23bd63de77e665fea41e18cdb68f999f0f4eefa3922af WatchSource:0}: Error finding container 543a4a6407b17214c7c23bd63de77e665fea41e18cdb68f999f0f4eefa3922af: Status 404 returned error can't find the container with id 543a4a6407b17214c7c23bd63de77e665fea41e18cdb68f999f0f4eefa3922af Sep 11 01:10:03.538043 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.538018 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d"] Sep 11 01:10:03.551409 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.551386 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" Sep 11 01:10:03.553663 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.553578 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-6ba087-serving-cert\"" Sep 11 01:10:03.555495 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.555449 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d"] Sep 11 01:10:03.609291 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.609265 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:03.635381 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.635356 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfblv\" (UniqueName: \"kubernetes.io/projected/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-kube-api-access-rfblv\") pod \"maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d\" (UID: \"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" Sep 11 01:10:03.635505 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.635393 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-metrics-certs\") pod \"maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d\" (UID: \"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" Sep 11 01:10:03.635505 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.635468 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d\" (UID: \"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" Sep 11 01:10:03.635618 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.635503 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d\" (UID: \"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" Sep 11 01:10:03.727061 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.727016 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:03.736937 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.736914 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d\" (UID: \"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" Sep 11 01:10:03.737045 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.736949 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d\" (UID: \"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" Sep 11 01:10:03.737045 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.736997 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfblv\" (UniqueName: \"kubernetes.io/projected/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-kube-api-access-rfblv\") pod \"maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d\" (UID: \"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" Sep 11 01:10:03.737045 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.737017 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-metrics-certs\") pod \"maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d\" (UID: \"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" Sep 11 01:10:03.737201 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:10:03.737076 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-serving-cert: secret "maas-api-e2e-worker-w0-6ba087-serving-cert" not found Sep 11 01:10:03.737201 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:10:03.737144 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-maas-api-tls podName:eaa119bd-3d65-4667-b2f3-50e7ac3dd00f nodeName:}" failed. No retries permitted until 2026-09-11 01:10:04.237123932 +0000 UTC m=+731.060577428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-maas-api-tls") pod "maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" (UID: "eaa119bd-3d65-4667-b2f3-50e7ac3dd00f") : secret "maas-api-e2e-worker-w0-6ba087-serving-cert" not found Sep 11 01:10:03.737811 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.737786 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d\" (UID: \"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" Sep 11 01:10:03.739567 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.739550 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-metrics-certs\") pod \"maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d\" (UID: \"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" Sep 11 01:10:03.756538 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.756512 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfblv\" (UniqueName: \"kubernetes.io/projected/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-kube-api-access-rfblv\") pod \"maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d\" (UID: \"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" Sep 11 01:10:03.832834 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.832749 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:03.852890 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.852831 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" event={"ID":"07f53eed-c8ba-42ee-ac68-f1d1cd2594db","Type":"ContainerStarted","Data":"543a4a6407b17214c7c23bd63de77e665fea41e18cdb68f999f0f4eefa3922af"} Sep 11 01:10:03.854410 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.854384 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg" event={"ID":"15915139-3495-41f1-b135-86fd42bf424e","Type":"ContainerStarted","Data":"03c601a891760afb6979d2e0c8b45c5ac2406fb072c745d3ce2c494498ae5af3"} Sep 11 01:10:03.854527 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.854426 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg" event={"ID":"15915139-3495-41f1-b135-86fd42bf424e","Type":"ContainerStarted","Data":"0c66cfe5bf4203bf8b85d4f9a76fec8cf29c9ebfa827dcc3f9e938fe16114b69"} Sep 11 01:10:03.854603 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.854529 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg" Sep 11 01:10:03.888472 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:03.888431 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg" podStartSLOduration=1.4473113450000001 podStartE2EDuration="1.888418379s" totalImagesPullingTime="441.107034ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:10:03.887719769 +0000 UTC m=+730.711173279" watchObservedRunningTime="2026-09-11 01:10:03.888418379 +0000 UTC m=+730.711871889" Sep 11 01:10:04.194933 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:04.194889 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:04.243232 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:04.243206 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d\" (UID: \"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" Sep 11 01:10:04.245630 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:04.245606 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d\" (UID: \"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" Sep 11 01:10:04.286984 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:04.286949 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:04.482932 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:04.482838 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" Sep 11 01:10:04.670455 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:04.670423 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:04.776673 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:04.776652 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d"] Sep 11 01:10:04.778103 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:10:04.778071 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa119bd_3d65_4667_b2f3_50e7ac3dd00f.slice/crio-9225834b37acaf3b96a1a692074595d79910dd2dfa64468b96a5ffdda2784261 WatchSource:0}: Error finding container 9225834b37acaf3b96a1a692074595d79910dd2dfa64468b96a5ffdda2784261: Status 404 returned error can't find the container with id 9225834b37acaf3b96a1a692074595d79910dd2dfa64468b96a5ffdda2784261 Sep 11 01:10:04.781541 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:04.781513 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:04.860477 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:04.860444 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" event={"ID":"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f","Type":"ContainerStarted","Data":"9225834b37acaf3b96a1a692074595d79910dd2dfa64468b96a5ffdda2784261"} Sep 11 01:10:04.862001 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:04.861977 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" event={"ID":"07f53eed-c8ba-42ee-ac68-f1d1cd2594db","Type":"ContainerStarted","Data":"8996366c5eb110a5110eb1bb69a4bdc986c1b8539bd365fd3430271c17860e59"} Sep 11 01:10:05.020743 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:05.020692 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:05.491717 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:05.491628 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:05.577185 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:05.577151 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:05.867954 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:05.867866 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" event={"ID":"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f","Type":"ContainerStarted","Data":"15454a02b38559e09a1e95352d6775a3a78b049aa471b98269dd1469439001ae"} Sep 11 01:10:05.868351 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:05.867952 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" Sep 11 01:10:05.868351 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:05.868124 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" Sep 11 01:10:05.890161 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:05.890111 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" podStartSLOduration=2.478321849 podStartE2EDuration="2.890093605s" totalImagesPullingTime="411.771756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:10:05.889536802 +0000 UTC m=+732.712990305" watchObservedRunningTime="2026-09-11 01:10:05.890093605 +0000 UTC m=+732.713547116" Sep 11 01:10:05.890566 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:05.890540 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" podStartSLOduration=4.468079302 podStartE2EDuration="4.890531938s" totalImagesPullingTime="422.452636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:10:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:10:04.882432469 +0000 UTC m=+731.705885979" watchObservedRunningTime="2026-09-11 01:10:05.890531938 +0000 UTC m=+732.713985449" Sep 11 01:10:06.278820 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:06.278786 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:06.372904 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:06.372870 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:06.515128 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:06.515101 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:06.656387 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:06.656308 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:06.860967 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:06.860936 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:06.948157 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:06.948127 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:08.377093 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:08.377056 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:08.471123 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:08.471083 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:09.705389 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:09.705357 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5"] Sep 11 01:10:09.711044 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:09.711019 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5" Sep 11 01:10:09.713597 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:09.713575 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-dc7dcd-serving-cert\"" Sep 11 01:10:09.721598 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:09.721573 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5"] Sep 11 01:10:09.743868 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:09.743816 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:09.792108 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:09.792069 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/897ee5bb-08dc-4d02-b571-f022f8c8de14-metrics-certs\") pod \"maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5\" (UID: \"897ee5bb-08dc-4d02-b571-f022f8c8de14\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5" Sep 11 01:10:09.792250 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:09.792133 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkvzs\" (UniqueName: \"kubernetes.io/projected/897ee5bb-08dc-4d02-b571-f022f8c8de14-kube-api-access-zkvzs\") pod \"maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5\" (UID: \"897ee5bb-08dc-4d02-b571-f022f8c8de14\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5" Sep 11 01:10:09.792250 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:09.792152 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/897ee5bb-08dc-4d02-b571-f022f8c8de14-maas-api-tls\") pod \"maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5\" (UID: \"897ee5bb-08dc-4d02-b571-f022f8c8de14\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5" Sep 11 01:10:09.792250 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:09.792180 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/897ee5bb-08dc-4d02-b571-f022f8c8de14-openshift-service-ca\") pod \"maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5\" (UID: \"897ee5bb-08dc-4d02-b571-f022f8c8de14\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5" Sep 11 01:10:09.842719 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:09.842680 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:09.873872 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:09.873834 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg" Sep 11 01:10:09.892762 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:09.892727 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkvzs\" (UniqueName: \"kubernetes.io/projected/897ee5bb-08dc-4d02-b571-f022f8c8de14-kube-api-access-zkvzs\") pod \"maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5\" (UID: \"897ee5bb-08dc-4d02-b571-f022f8c8de14\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5" Sep 11 01:10:09.893004 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:09.892985 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/897ee5bb-08dc-4d02-b571-f022f8c8de14-maas-api-tls\") pod \"maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5\" (UID: \"897ee5bb-08dc-4d02-b571-f022f8c8de14\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5" Sep 11 01:10:09.893154 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:09.893141 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/897ee5bb-08dc-4d02-b571-f022f8c8de14-openshift-service-ca\") pod \"maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5\" (UID: \"897ee5bb-08dc-4d02-b571-f022f8c8de14\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5" Sep 11 01:10:09.893341 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:09.893326 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/897ee5bb-08dc-4d02-b571-f022f8c8de14-metrics-certs\") pod \"maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5\" (UID: \"897ee5bb-08dc-4d02-b571-f022f8c8de14\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5" Sep 11 01:10:09.895628 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:09.895602 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/897ee5bb-08dc-4d02-b571-f022f8c8de14-openshift-service-ca\") pod \"maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5\" (UID: \"897ee5bb-08dc-4d02-b571-f022f8c8de14\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5" Sep 11 01:10:09.900326 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:09.900302 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/897ee5bb-08dc-4d02-b571-f022f8c8de14-metrics-certs\") pod \"maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5\" (UID: \"897ee5bb-08dc-4d02-b571-f022f8c8de14\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5" Sep 11 01:10:09.908904 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:09.908877 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/897ee5bb-08dc-4d02-b571-f022f8c8de14-maas-api-tls\") pod \"maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5\" (UID: \"897ee5bb-08dc-4d02-b571-f022f8c8de14\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5" Sep 11 01:10:09.911494 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:09.911467 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkvzs\" (UniqueName: \"kubernetes.io/projected/897ee5bb-08dc-4d02-b571-f022f8c8de14-kube-api-access-zkvzs\") pod \"maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5\" (UID: \"897ee5bb-08dc-4d02-b571-f022f8c8de14\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5" Sep 11 01:10:09.962381 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:09.962294 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:10.023766 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:10.023736 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5" Sep 11 01:10:10.059548 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:10.059508 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:10.297402 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:10.297380 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5"] Sep 11 01:10:10.299749 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:10:10.299723 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod897ee5bb_08dc_4d02_b571_f022f8c8de14.slice/crio-9f3fe5786f8ef7b3c3f18626a0abd368dde07cbd943c5f12e5f5a6dca313e3fc WatchSource:0}: Error finding container 9f3fe5786f8ef7b3c3f18626a0abd368dde07cbd943c5f12e5f5a6dca313e3fc: Status 404 returned error can't find the container with id 9f3fe5786f8ef7b3c3f18626a0abd368dde07cbd943c5f12e5f5a6dca313e3fc Sep 11 01:10:10.388928 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:10.388897 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:10.483549 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:10.483516 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:10.896344 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:10.896307 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5" event={"ID":"897ee5bb-08dc-4d02-b571-f022f8c8de14","Type":"ContainerStarted","Data":"1ba816af8343911dd36d46c5f08ce6f3940418740095c6a453a8ff7f931df9e4"} Sep 11 01:10:10.896661 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:10.896363 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5" event={"ID":"897ee5bb-08dc-4d02-b571-f022f8c8de14","Type":"ContainerStarted","Data":"9f3fe5786f8ef7b3c3f18626a0abd368dde07cbd943c5f12e5f5a6dca313e3fc"} Sep 11 01:10:10.896661 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:10.896393 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5" Sep 11 01:10:10.923214 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:10.923171 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5" podStartSLOduration=1.487494334 podStartE2EDuration="1.923154556s" totalImagesPullingTime="435.660222ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:10:10.922486747 +0000 UTC m=+737.745940257" watchObservedRunningTime="2026-09-11 01:10:10.923154556 +0000 UTC m=+737.746608066" Sep 11 01:10:11.414422 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:11.414378 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr"] Sep 11 01:10:11.418383 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:11.418356 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr" Sep 11 01:10:11.420602 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:11.420573 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0bc9acd8-serving-cert\"" Sep 11 01:10:11.430694 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:11.430667 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr"] Sep 11 01:10:11.451309 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:11.451278 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:11.507202 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:11.507175 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/934affd6-d045-4e79-8319-f0e5cd98556c-maas-api-tls\") pod \"maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr\" (UID: \"934affd6-d045-4e79-8319-f0e5cd98556c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr" Sep 11 01:10:11.507320 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:11.507229 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/934affd6-d045-4e79-8319-f0e5cd98556c-metrics-certs\") pod \"maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr\" (UID: \"934affd6-d045-4e79-8319-f0e5cd98556c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr" Sep 11 01:10:11.507320 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:11.507284 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/934affd6-d045-4e79-8319-f0e5cd98556c-openshift-service-ca\") pod \"maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr\" (UID: \"934affd6-d045-4e79-8319-f0e5cd98556c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr" Sep 11 01:10:11.507413 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:11.507361 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8f52\" (UniqueName: \"kubernetes.io/projected/934affd6-d045-4e79-8319-f0e5cd98556c-kube-api-access-s8f52\") pod \"maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr\" (UID: \"934affd6-d045-4e79-8319-f0e5cd98556c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr" Sep 11 01:10:11.517045 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:11.517015 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:11.608519 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:11.608497 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/934affd6-d045-4e79-8319-f0e5cd98556c-maas-api-tls\") pod \"maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr\" (UID: \"934affd6-d045-4e79-8319-f0e5cd98556c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr" Sep 11 01:10:11.608635 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:11.608543 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/934affd6-d045-4e79-8319-f0e5cd98556c-metrics-certs\") pod \"maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr\" (UID: \"934affd6-d045-4e79-8319-f0e5cd98556c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr" Sep 11 01:10:11.608635 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:11.608564 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/934affd6-d045-4e79-8319-f0e5cd98556c-openshift-service-ca\") pod \"maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr\" (UID: \"934affd6-d045-4e79-8319-f0e5cd98556c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr" Sep 11 01:10:11.608747 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:11.608682 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8f52\" (UniqueName: \"kubernetes.io/projected/934affd6-d045-4e79-8319-f0e5cd98556c-kube-api-access-s8f52\") pod \"maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr\" (UID: \"934affd6-d045-4e79-8319-f0e5cd98556c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr" Sep 11 01:10:11.609149 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:11.609131 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/934affd6-d045-4e79-8319-f0e5cd98556c-openshift-service-ca\") pod \"maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr\" (UID: \"934affd6-d045-4e79-8319-f0e5cd98556c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr" Sep 11 01:10:11.611018 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:11.610996 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/934affd6-d045-4e79-8319-f0e5cd98556c-maas-api-tls\") pod \"maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr\" (UID: \"934affd6-d045-4e79-8319-f0e5cd98556c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr" Sep 11 01:10:11.611101 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:11.611064 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/934affd6-d045-4e79-8319-f0e5cd98556c-metrics-certs\") pod \"maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr\" (UID: \"934affd6-d045-4e79-8319-f0e5cd98556c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr" Sep 11 01:10:11.616242 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:11.616217 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8f52\" (UniqueName: \"kubernetes.io/projected/934affd6-d045-4e79-8319-f0e5cd98556c-kube-api-access-s8f52\") pod \"maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr\" (UID: \"934affd6-d045-4e79-8319-f0e5cd98556c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr" Sep 11 01:10:11.617458 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:11.617435 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:11.731300 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:11.731271 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr" Sep 11 01:10:11.878964 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:11.878939 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" Sep 11 01:10:11.879437 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:11.879419 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" Sep 11 01:10:11.956720 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:11.956684 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:12.009301 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:12.009275 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr"] Sep 11 01:10:12.010954 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:10:12.010923 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod934affd6_d045_4e79_8319_f0e5cd98556c.slice/crio-87c1eea5491bf59a4ea4176f53364abc371cd670b4fe4c310f8f0400b3ebb45d WatchSource:0}: Error finding container 87c1eea5491bf59a4ea4176f53364abc371cd670b4fe4c310f8f0400b3ebb45d: Status 404 returned error can't find the container with id 87c1eea5491bf59a4ea4176f53364abc371cd670b4fe4c310f8f0400b3ebb45d Sep 11 01:10:12.053086 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:12.053054 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:12.358751 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:12.358674 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:12.908276 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:12.908240 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr" event={"ID":"934affd6-d045-4e79-8319-f0e5cd98556c","Type":"ContainerStarted","Data":"cef8802f99663223afe41c6eb120938ef50264f4fdc35840fa450f08214d505f"} Sep 11 01:10:12.908437 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:12.908291 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr" event={"ID":"934affd6-d045-4e79-8319-f0e5cd98556c","Type":"ContainerStarted","Data":"87c1eea5491bf59a4ea4176f53364abc371cd670b4fe4c310f8f0400b3ebb45d"} Sep 11 01:10:12.908437 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:12.908314 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr" Sep 11 01:10:12.926740 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:12.926694 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr" podStartSLOduration=1.56227312 podStartE2EDuration="1.926679188s" totalImagesPullingTime="364.406068ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:10:12.924897433 +0000 UTC m=+739.748350943" watchObservedRunningTime="2026-09-11 01:10:12.926679188 +0000 UTC m=+739.750132764" Sep 11 01:10:13.220440 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:13.220404 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:15.460775 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:15.460734 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:15.646954 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:15.646918 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:15.761138 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:15.761068 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:15.866087 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:15.866054 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:16.907192 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:16.907164 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5" Sep 11 01:10:18.512362 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:18.512332 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:18.918004 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:18.917930 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr" Sep 11 01:10:20.148770 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:20.148730 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:22.047806 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:22.047774 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:22.146502 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:22.146475 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:22.283895 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:22.283864 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:22.623060 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:22.623010 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:23.925690 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:23.925652 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:24.630755 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:24.630721 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:25.090066 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:25.090030 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:25.190266 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:25.190224 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:25.438594 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:25.438558 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:25.539382 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:25.539333 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:25.641133 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:25.641101 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:25.728357 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:25.728239 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:25.797718 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:25.797681 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:26.073266 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:26.073178 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:26.289576 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:26.289544 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:28.297577 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:28.297535 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:28.365810 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:28.365775 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:28.447095 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:28.447066 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:28.548112 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:28.548035 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:28.662663 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:28.662626 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:28.759556 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:28.759518 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:29.750643 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:29.750609 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2"] Sep 11 01:10:29.755025 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:29.755002 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2" Sep 11 01:10:29.758241 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:29.758110 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-891e67-serving-cert\"" Sep 11 01:10:29.761859 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:29.761826 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2"] Sep 11 01:10:29.794910 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:29.794885 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:29.873283 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:29.873258 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw64m\" (UniqueName: \"kubernetes.io/projected/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b-kube-api-access-cw64m\") pod \"maas-api-e2e-inf-b-891e67-8485698886-nmbg2\" (UID: \"9356eb13-4c12-4d62-b8f7-d0ec958fcc4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2" Sep 11 01:10:29.873399 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:29.873308 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b-maas-api-tls\") pod \"maas-api-e2e-inf-b-891e67-8485698886-nmbg2\" (UID: \"9356eb13-4c12-4d62-b8f7-d0ec958fcc4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2" Sep 11 01:10:29.873399 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:29.873332 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-891e67-8485698886-nmbg2\" (UID: \"9356eb13-4c12-4d62-b8f7-d0ec958fcc4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2" Sep 11 01:10:29.873399 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:29.873378 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b-metrics-certs\") pod \"maas-api-e2e-inf-b-891e67-8485698886-nmbg2\" (UID: \"9356eb13-4c12-4d62-b8f7-d0ec958fcc4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2" Sep 11 01:10:29.897544 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:29.897521 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:29.973800 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:29.973777 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw64m\" (UniqueName: \"kubernetes.io/projected/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b-kube-api-access-cw64m\") pod \"maas-api-e2e-inf-b-891e67-8485698886-nmbg2\" (UID: \"9356eb13-4c12-4d62-b8f7-d0ec958fcc4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2" Sep 11 01:10:29.973923 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:29.973822 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b-maas-api-tls\") pod \"maas-api-e2e-inf-b-891e67-8485698886-nmbg2\" (UID: \"9356eb13-4c12-4d62-b8f7-d0ec958fcc4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2" Sep 11 01:10:29.973923 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:29.973861 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-891e67-8485698886-nmbg2\" (UID: \"9356eb13-4c12-4d62-b8f7-d0ec958fcc4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2" Sep 11 01:10:29.973923 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:29.973886 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b-metrics-certs\") pod \"maas-api-e2e-inf-b-891e67-8485698886-nmbg2\" (UID: \"9356eb13-4c12-4d62-b8f7-d0ec958fcc4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2" Sep 11 01:10:29.974557 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:29.974539 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-891e67-8485698886-nmbg2\" (UID: \"9356eb13-4c12-4d62-b8f7-d0ec958fcc4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2" Sep 11 01:10:29.976141 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:29.976125 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b-maas-api-tls\") pod \"maas-api-e2e-inf-b-891e67-8485698886-nmbg2\" (UID: \"9356eb13-4c12-4d62-b8f7-d0ec958fcc4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2" Sep 11 01:10:29.976689 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:29.976673 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b-metrics-certs\") pod \"maas-api-e2e-inf-b-891e67-8485698886-nmbg2\" (UID: \"9356eb13-4c12-4d62-b8f7-d0ec958fcc4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2" Sep 11 01:10:29.981723 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:29.981700 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw64m\" (UniqueName: \"kubernetes.io/projected/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b-kube-api-access-cw64m\") pod \"maas-api-e2e-inf-b-891e67-8485698886-nmbg2\" (UID: \"9356eb13-4c12-4d62-b8f7-d0ec958fcc4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2" Sep 11 01:10:29.995721 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:29.995699 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:30.070783 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:30.070728 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2" Sep 11 01:10:30.342386 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:30.342286 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2"] Sep 11 01:10:30.345664 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:10:30.345610 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9356eb13_4c12_4d62_b8f7_d0ec958fcc4b.slice/crio-6d0166049f365b4ad5654c808eb8cefe152afcdd233b241f92cebe76ffb0010d WatchSource:0}: Error finding container 6d0166049f365b4ad5654c808eb8cefe152afcdd233b241f92cebe76ffb0010d: Status 404 returned error can't find the container with id 6d0166049f365b4ad5654c808eb8cefe152afcdd233b241f92cebe76ffb0010d Sep 11 01:10:30.396416 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:30.396384 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:30.482658 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:30.482630 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:30.902323 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:30.902291 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:31.000735 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:31.000703 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:31.002443 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:31.002414 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2" event={"ID":"9356eb13-4c12-4d62-b8f7-d0ec958fcc4b","Type":"ContainerStarted","Data":"22ac1554a4f60f14b0dab08ff7676688baf2416f040acdd4a95c3bb99295cb49"} Sep 11 01:10:31.002564 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:31.002463 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2" event={"ID":"9356eb13-4c12-4d62-b8f7-d0ec958fcc4b","Type":"ContainerStarted","Data":"6d0166049f365b4ad5654c808eb8cefe152afcdd233b241f92cebe76ffb0010d"} Sep 11 01:10:31.002564 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:31.002520 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2" Sep 11 01:10:31.021188 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:31.021147 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2" podStartSLOduration=1.547289403 podStartE2EDuration="2.021132801s" totalImagesPullingTime="473.843398ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:10:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:10:31.019474012 +0000 UTC m=+757.842927521" watchObservedRunningTime="2026-09-11 01:10:31.021132801 +0000 UTC m=+757.844586310" Sep 11 01:10:31.101179 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:31.101112 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:31.531260 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:31.531216 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:32.304491 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:32.304452 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:33.239381 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:33.239327 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:33.338098 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:33.338067 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:33.977959 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:33.977922 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:34.651530 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:34.651490 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:34.753574 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:34.753539 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:37.012317 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:37.012291 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2" Sep 11 01:10:50.763990 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:50.763950 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:51.427430 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:51.427397 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:52.087648 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:52.087615 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:52.632980 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:52.632949 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr"] Sep 11 01:10:52.633192 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:52.633160 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr" podUID="934affd6-d045-4e79-8319-f0e5cd98556c" containerName="maas-api" containerID="cri-o://cef8802f99663223afe41c6eb120938ef50264f4fdc35840fa450f08214d505f" gracePeriod=30 Sep 11 01:10:52.666874 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:52.666832 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:52.739550 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:52.739493 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:52.794535 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:52.794505 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:52.885063 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:52.885010 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr" Sep 11 01:10:52.894228 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:52.894208 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:52.978446 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:52.978416 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:52.982892 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:52.982870 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/934affd6-d045-4e79-8319-f0e5cd98556c-openshift-service-ca\") pod \"934affd6-d045-4e79-8319-f0e5cd98556c\" (UID: \"934affd6-d045-4e79-8319-f0e5cd98556c\") " Sep 11 01:10:52.982892 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:52.982905 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/934affd6-d045-4e79-8319-f0e5cd98556c-metrics-certs\") pod \"934affd6-d045-4e79-8319-f0e5cd98556c\" (UID: \"934affd6-d045-4e79-8319-f0e5cd98556c\") " Sep 11 01:10:52.983102 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:52.982944 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8f52\" (UniqueName: \"kubernetes.io/projected/934affd6-d045-4e79-8319-f0e5cd98556c-kube-api-access-s8f52\") pod \"934affd6-d045-4e79-8319-f0e5cd98556c\" (UID: \"934affd6-d045-4e79-8319-f0e5cd98556c\") " Sep 11 01:10:52.983102 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:52.983052 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/934affd6-d045-4e79-8319-f0e5cd98556c-maas-api-tls\") pod \"934affd6-d045-4e79-8319-f0e5cd98556c\" (UID: \"934affd6-d045-4e79-8319-f0e5cd98556c\") " Sep 11 01:10:52.983247 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:52.983226 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/934affd6-d045-4e79-8319-f0e5cd98556c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "934affd6-d045-4e79-8319-f0e5cd98556c" (UID: "934affd6-d045-4e79-8319-f0e5cd98556c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:10:52.983368 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:52.983349 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/934affd6-d045-4e79-8319-f0e5cd98556c-openshift-service-ca\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:10:52.986583 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:52.986550 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/934affd6-d045-4e79-8319-f0e5cd98556c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "934affd6-d045-4e79-8319-f0e5cd98556c" (UID: "934affd6-d045-4e79-8319-f0e5cd98556c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:10:52.987989 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:52.987952 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/934affd6-d045-4e79-8319-f0e5cd98556c-kube-api-access-s8f52" (OuterVolumeSpecName: "kube-api-access-s8f52") pod "934affd6-d045-4e79-8319-f0e5cd98556c" (UID: "934affd6-d045-4e79-8319-f0e5cd98556c"). InnerVolumeSpecName "kube-api-access-s8f52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:10:52.988332 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:52.988310 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/934affd6-d045-4e79-8319-f0e5cd98556c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "934affd6-d045-4e79-8319-f0e5cd98556c" (UID: "934affd6-d045-4e79-8319-f0e5cd98556c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:10:53.041178 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:53.041148 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:53.084706 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:53.084664 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/934affd6-d045-4e79-8319-f0e5cd98556c-maas-api-tls\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:10:53.084706 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:53.084685 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/934affd6-d045-4e79-8319-f0e5cd98556c-metrics-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:10:53.084706 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:53.084695 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8f52\" (UniqueName: \"kubernetes.io/projected/934affd6-d045-4e79-8319-f0e5cd98556c-kube-api-access-s8f52\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:10:53.117467 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:53.117437 2554 generic.go:356] "Generic (PLEG): container finished" podID="934affd6-d045-4e79-8319-f0e5cd98556c" containerID="cef8802f99663223afe41c6eb120938ef50264f4fdc35840fa450f08214d505f" exitCode=0 Sep 11 01:10:53.117754 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:53.117495 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr" event={"ID":"934affd6-d045-4e79-8319-f0e5cd98556c","Type":"ContainerDied","Data":"cef8802f99663223afe41c6eb120938ef50264f4fdc35840fa450f08214d505f"} Sep 11 01:10:53.117754 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:53.117537 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr" event={"ID":"934affd6-d045-4e79-8319-f0e5cd98556c","Type":"ContainerDied","Data":"87c1eea5491bf59a4ea4176f53364abc371cd670b4fe4c310f8f0400b3ebb45d"} Sep 11 01:10:53.117754 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:53.117561 2554 scope.go:117] "RemoveContainer" containerID="cef8802f99663223afe41c6eb120938ef50264f4fdc35840fa450f08214d505f" Sep 11 01:10:53.117754 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:53.117499 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr" Sep 11 01:10:53.126601 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:53.126586 2554 scope.go:117] "RemoveContainer" containerID="cef8802f99663223afe41c6eb120938ef50264f4fdc35840fa450f08214d505f" Sep 11 01:10:53.126881 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:10:53.126856 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cef8802f99663223afe41c6eb120938ef50264f4fdc35840fa450f08214d505f\": container with ID starting with cef8802f99663223afe41c6eb120938ef50264f4fdc35840fa450f08214d505f not found: ID does not exist" containerID="cef8802f99663223afe41c6eb120938ef50264f4fdc35840fa450f08214d505f" Sep 11 01:10:53.126954 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:53.126892 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cef8802f99663223afe41c6eb120938ef50264f4fdc35840fa450f08214d505f"} err="failed to get container status \"cef8802f99663223afe41c6eb120938ef50264f4fdc35840fa450f08214d505f\": rpc error: code = NotFound desc = could not find container \"cef8802f99663223afe41c6eb120938ef50264f4fdc35840fa450f08214d505f\": container with ID starting with cef8802f99663223afe41c6eb120938ef50264f4fdc35840fa450f08214d505f not found: ID does not exist" Sep 11 01:10:53.138237 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:53.138190 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr"] Sep 11 01:10:53.143937 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:53.143915 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0bc9acd8-7d8b95bcb8-tt6kr"] Sep 11 01:10:53.515869 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:53.515798 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:53.619427 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:53.619392 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:53.679087 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:53.679059 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="934affd6-d045-4e79-8319-f0e5cd98556c" path="/var/lib/kubelet/pods/934affd6-d045-4e79-8319-f0e5cd98556c/volumes" Sep 11 01:10:53.775762 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:53.775693 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:54.050523 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:54.050433 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:54.905494 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:54.905457 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:55.401155 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.401126 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv"] Sep 11 01:10:55.401535 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.401523 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="934affd6-d045-4e79-8319-f0e5cd98556c" containerName="maas-api" Sep 11 01:10:55.401579 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.401537 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="934affd6-d045-4e79-8319-f0e5cd98556c" containerName="maas-api" Sep 11 01:10:55.401619 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.401604 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="934affd6-d045-4e79-8319-f0e5cd98556c" containerName="maas-api" Sep 11 01:10:55.406118 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.406099 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:10:55.411453 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.411422 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-651569\"/\"kube-root-ca.crt\"" Sep 11 01:10:55.412442 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.412418 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-651569\"/\"default-dockercfg-cdvkt\"" Sep 11 01:10:55.412556 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.412499 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-651569\"/\"simulator-w4-651569-kserve-self-signed-certs\"" Sep 11 01:10:55.412556 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.412517 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-651569\"/\"openshift-service-ca.crt\"" Sep 11 01:10:55.430407 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.430388 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv"] Sep 11 01:10:55.505045 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.505019 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74c8f859-e529-4fe1-91f4-39f9a1be60fa-tmp-dir\") pod \"simulator-w4-651569-kserve-744844544c-sb5xv\" (UID: \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\") " pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:10:55.505150 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.505053 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74c8f859-e529-4fe1-91f4-39f9a1be60fa-home\") pod \"simulator-w4-651569-kserve-744844544c-sb5xv\" (UID: \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\") " pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:10:55.505150 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.505092 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74c8f859-e529-4fe1-91f4-39f9a1be60fa-dshm\") pod \"simulator-w4-651569-kserve-744844544c-sb5xv\" (UID: \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\") " pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:10:55.505150 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.505137 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfk2v\" (UniqueName: \"kubernetes.io/projected/74c8f859-e529-4fe1-91f4-39f9a1be60fa-kube-api-access-nfk2v\") pod \"simulator-w4-651569-kserve-744844544c-sb5xv\" (UID: \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\") " pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:10:55.505305 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.505158 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74c8f859-e529-4fe1-91f4-39f9a1be60fa-tls-certs\") pod \"simulator-w4-651569-kserve-744844544c-sb5xv\" (UID: \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\") " pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:10:55.505305 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.505238 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74c8f859-e529-4fe1-91f4-39f9a1be60fa-model-cache\") pod \"simulator-w4-651569-kserve-744844544c-sb5xv\" (UID: \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\") " pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:10:55.605567 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.605537 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74c8f859-e529-4fe1-91f4-39f9a1be60fa-tmp-dir\") pod \"simulator-w4-651569-kserve-744844544c-sb5xv\" (UID: \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\") " pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:10:55.605567 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.605568 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74c8f859-e529-4fe1-91f4-39f9a1be60fa-home\") pod \"simulator-w4-651569-kserve-744844544c-sb5xv\" (UID: \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\") " pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:10:55.605759 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.605600 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74c8f859-e529-4fe1-91f4-39f9a1be60fa-dshm\") pod \"simulator-w4-651569-kserve-744844544c-sb5xv\" (UID: \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\") " pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:10:55.605759 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.605638 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfk2v\" (UniqueName: \"kubernetes.io/projected/74c8f859-e529-4fe1-91f4-39f9a1be60fa-kube-api-access-nfk2v\") pod \"simulator-w4-651569-kserve-744844544c-sb5xv\" (UID: \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\") " pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:10:55.605759 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.605655 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74c8f859-e529-4fe1-91f4-39f9a1be60fa-tls-certs\") pod \"simulator-w4-651569-kserve-744844544c-sb5xv\" (UID: \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\") " pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:10:55.605759 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.605676 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74c8f859-e529-4fe1-91f4-39f9a1be60fa-model-cache\") pod \"simulator-w4-651569-kserve-744844544c-sb5xv\" (UID: \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\") " pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:10:55.606012 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.605941 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74c8f859-e529-4fe1-91f4-39f9a1be60fa-home\") pod \"simulator-w4-651569-kserve-744844544c-sb5xv\" (UID: \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\") " pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:10:55.606012 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.605980 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74c8f859-e529-4fe1-91f4-39f9a1be60fa-tmp-dir\") pod \"simulator-w4-651569-kserve-744844544c-sb5xv\" (UID: \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\") " pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:10:55.606126 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.606088 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74c8f859-e529-4fe1-91f4-39f9a1be60fa-model-cache\") pod \"simulator-w4-651569-kserve-744844544c-sb5xv\" (UID: \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\") " pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:10:55.608131 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.608110 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74c8f859-e529-4fe1-91f4-39f9a1be60fa-dshm\") pod \"simulator-w4-651569-kserve-744844544c-sb5xv\" (UID: \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\") " pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:10:55.608430 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.608411 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74c8f859-e529-4fe1-91f4-39f9a1be60fa-tls-certs\") pod \"simulator-w4-651569-kserve-744844544c-sb5xv\" (UID: \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\") " pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:10:55.614136 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.614116 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfk2v\" (UniqueName: \"kubernetes.io/projected/74c8f859-e529-4fe1-91f4-39f9a1be60fa-kube-api-access-nfk2v\") pod \"simulator-w4-651569-kserve-744844544c-sb5xv\" (UID: \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\") " pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:10:55.696541 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.696502 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:55.714709 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.714689 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:10:55.999239 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:55.999209 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv"] Sep 11 01:10:56.000977 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:10:56.000952 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74c8f859_e529_4fe1_91f4_39f9a1be60fa.slice/crio-c69c9837e0c93d8676df847130818bc098b2a0f3bd7dae8f12425b7cf9485fcb WatchSource:0}: Error finding container c69c9837e0c93d8676df847130818bc098b2a0f3bd7dae8f12425b7cf9485fcb: Status 404 returned error can't find the container with id c69c9837e0c93d8676df847130818bc098b2a0f3bd7dae8f12425b7cf9485fcb Sep 11 01:10:56.135706 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:56.135680 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" event={"ID":"74c8f859-e529-4fe1-91f4-39f9a1be60fa","Type":"ContainerStarted","Data":"2a958cebc5b05181555778dc00bc7b60f973bf1328f0de711d309b42d54a921b"} Sep 11 01:10:56.135860 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:56.135716 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" event={"ID":"74c8f859-e529-4fe1-91f4-39f9a1be60fa","Type":"ContainerStarted","Data":"c69c9837e0c93d8676df847130818bc098b2a0f3bd7dae8f12425b7cf9485fcb"} Sep 11 01:10:56.151075 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:56.151028 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" podStartSLOduration=1.151013172 podStartE2EDuration="1.151013172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:10:56.150023455 +0000 UTC m=+782.973476988" watchObservedRunningTime="2026-09-11 01:10:56.151013172 +0000 UTC m=+782.974466733" Sep 11 01:10:56.955278 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:56.955241 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:57.055914 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:57.055879 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:57.177711 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:57.177681 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx"] Sep 11 01:10:57.181515 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:57.181490 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" Sep 11 01:10:57.183654 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:57.183630 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-1d3c7cc9-serving-cert\"" Sep 11 01:10:57.192305 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:57.192277 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx"] Sep 11 01:10:57.208555 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:57.208487 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:57.272034 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:57.272004 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:57.322898 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:57.322875 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-openshift-service-ca\") pod \"maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx\" (UID: \"90d99681-7ef6-4f8e-99dd-9c37e7db47eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" Sep 11 01:10:57.322983 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:57.322939 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-maas-api-tls\") pod \"maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx\" (UID: \"90d99681-7ef6-4f8e-99dd-9c37e7db47eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" Sep 11 01:10:57.322983 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:57.322957 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2wnm\" (UniqueName: \"kubernetes.io/projected/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-kube-api-access-f2wnm\") pod \"maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx\" (UID: \"90d99681-7ef6-4f8e-99dd-9c37e7db47eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" Sep 11 01:10:57.323095 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:57.322983 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-metrics-certs\") pod \"maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx\" (UID: \"90d99681-7ef6-4f8e-99dd-9c37e7db47eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" Sep 11 01:10:57.424126 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:57.424102 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-metrics-certs\") pod \"maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx\" (UID: \"90d99681-7ef6-4f8e-99dd-9c37e7db47eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" Sep 11 01:10:57.424229 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:57.424169 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-openshift-service-ca\") pod \"maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx\" (UID: \"90d99681-7ef6-4f8e-99dd-9c37e7db47eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" Sep 11 01:10:57.424294 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:57.424228 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-maas-api-tls\") pod \"maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx\" (UID: \"90d99681-7ef6-4f8e-99dd-9c37e7db47eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" Sep 11 01:10:57.424294 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:57.424250 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2wnm\" (UniqueName: \"kubernetes.io/projected/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-kube-api-access-f2wnm\") pod \"maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx\" (UID: \"90d99681-7ef6-4f8e-99dd-9c37e7db47eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" Sep 11 01:10:57.424415 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:10:57.424308 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-serving-cert: secret "maas-api-e2e-migrate-1d3c7cc9-serving-cert" not found Sep 11 01:10:57.424415 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:10:57.424379 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-maas-api-tls podName:90d99681-7ef6-4f8e-99dd-9c37e7db47eb nodeName:}" failed. No retries permitted until 2026-09-11 01:10:57.924363501 +0000 UTC m=+784.747816990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-maas-api-tls") pod "maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" (UID: "90d99681-7ef6-4f8e-99dd-9c37e7db47eb") : secret "maas-api-e2e-migrate-1d3c7cc9-serving-cert" not found Sep 11 01:10:57.424802 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:57.424782 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-openshift-service-ca\") pod \"maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx\" (UID: \"90d99681-7ef6-4f8e-99dd-9c37e7db47eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" Sep 11 01:10:57.426543 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:57.426524 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-metrics-certs\") pod \"maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx\" (UID: \"90d99681-7ef6-4f8e-99dd-9c37e7db47eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" Sep 11 01:10:57.431864 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:57.431828 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2wnm\" (UniqueName: \"kubernetes.io/projected/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-kube-api-access-f2wnm\") pod \"maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx\" (UID: \"90d99681-7ef6-4f8e-99dd-9c37e7db47eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" Sep 11 01:10:57.682216 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:57.682175 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:57.781773 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:57.781739 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:57.865739 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:57.865704 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:10:57.928608 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:57.928571 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-maas-api-tls\") pod \"maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx\" (UID: \"90d99681-7ef6-4f8e-99dd-9c37e7db47eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" Sep 11 01:10:57.931148 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:57.931118 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-maas-api-tls\") pod \"maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx\" (UID: \"90d99681-7ef6-4f8e-99dd-9c37e7db47eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" Sep 11 01:10:58.096087 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:58.096015 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" Sep 11 01:10:58.372538 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:58.372510 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx"] Sep 11 01:10:58.374088 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:10:58.374057 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d99681_7ef6_4f8e_99dd_9c37e7db47eb.slice/crio-68349939366bb947da527c81b32114a59e5fc7f96ef421d9cfccd33b3fe531e4 WatchSource:0}: Error finding container 68349939366bb947da527c81b32114a59e5fc7f96ef421d9cfccd33b3fe531e4: Status 404 returned error can't find the container with id 68349939366bb947da527c81b32114a59e5fc7f96ef421d9cfccd33b3fe531e4 Sep 11 01:10:59.157158 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:59.157123 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" event={"ID":"90d99681-7ef6-4f8e-99dd-9c37e7db47eb","Type":"ContainerStarted","Data":"72026c8146ca4d900cd034b7b043dc30b618439301ffc3d267fddb6d2853ff8d"} Sep 11 01:10:59.157464 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:59.157188 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" Sep 11 01:10:59.157464 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:59.157206 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" event={"ID":"90d99681-7ef6-4f8e-99dd-9c37e7db47eb","Type":"ContainerStarted","Data":"68349939366bb947da527c81b32114a59e5fc7f96ef421d9cfccd33b3fe531e4"} Sep 11 01:10:59.175464 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:10:59.175422 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" podStartSLOduration=1.497777634 podStartE2EDuration="2.175406023s" totalImagesPullingTime="677.628389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:10:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:10:59.173929976 +0000 UTC m=+785.997383517" watchObservedRunningTime="2026-09-11 01:10:59.175406023 +0000 UTC m=+785.998859569" Sep 11 01:11:05.168015 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:05.167987 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" Sep 11 01:11:05.715508 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:05.715467 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:11:05.715686 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:05.715523 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:11:05.728273 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:05.728245 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:11:06.204734 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:06.204705 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:11:08.113273 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:08.113235 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:11:22.998145 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:22.998110 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2"] Sep 11 01:11:23.002392 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.002371 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:11:23.006007 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.005785 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6ba087\"/\"default-dockercfg-ld827\"" Sep 11 01:11:23.006007 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.005785 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6ba087\"/\"kube-root-ca.crt\"" Sep 11 01:11:23.006212 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.006132 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6ba087\"/\"simulator-w0-6ba087-kserve-self-signed-certs\"" Sep 11 01:11:23.006388 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.006364 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6ba087\"/\"openshift-service-ca.crt\"" Sep 11 01:11:23.008784 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.008759 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2"] Sep 11 01:11:23.024007 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.023979 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:11:23.060717 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.060692 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2738da06-3225-42c9-b77a-86e864a9babd-tls-certs\") pod \"simulator-w0-6ba087-kserve-5bbd44496f-cmpn2\" (UID: \"2738da06-3225-42c9-b77a-86e864a9babd\") " pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:11:23.060831 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.060771 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2738da06-3225-42c9-b77a-86e864a9babd-model-cache\") pod \"simulator-w0-6ba087-kserve-5bbd44496f-cmpn2\" (UID: \"2738da06-3225-42c9-b77a-86e864a9babd\") " pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:11:23.060831 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.060794 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmvt4\" (UniqueName: \"kubernetes.io/projected/2738da06-3225-42c9-b77a-86e864a9babd-kube-api-access-gmvt4\") pod \"simulator-w0-6ba087-kserve-5bbd44496f-cmpn2\" (UID: \"2738da06-3225-42c9-b77a-86e864a9babd\") " pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:11:23.060831 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.060816 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2738da06-3225-42c9-b77a-86e864a9babd-home\") pod \"simulator-w0-6ba087-kserve-5bbd44496f-cmpn2\" (UID: \"2738da06-3225-42c9-b77a-86e864a9babd\") " pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:11:23.060831 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.060830 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2738da06-3225-42c9-b77a-86e864a9babd-tmp-dir\") pod \"simulator-w0-6ba087-kserve-5bbd44496f-cmpn2\" (UID: \"2738da06-3225-42c9-b77a-86e864a9babd\") " pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:11:23.061059 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.060952 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2738da06-3225-42c9-b77a-86e864a9babd-dshm\") pod \"simulator-w0-6ba087-kserve-5bbd44496f-cmpn2\" (UID: \"2738da06-3225-42c9-b77a-86e864a9babd\") " pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:11:23.161481 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.161458 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2738da06-3225-42c9-b77a-86e864a9babd-model-cache\") pod \"simulator-w0-6ba087-kserve-5bbd44496f-cmpn2\" (UID: \"2738da06-3225-42c9-b77a-86e864a9babd\") " pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:11:23.161594 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.161490 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmvt4\" (UniqueName: \"kubernetes.io/projected/2738da06-3225-42c9-b77a-86e864a9babd-kube-api-access-gmvt4\") pod \"simulator-w0-6ba087-kserve-5bbd44496f-cmpn2\" (UID: \"2738da06-3225-42c9-b77a-86e864a9babd\") " pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:11:23.161594 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.161510 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2738da06-3225-42c9-b77a-86e864a9babd-home\") pod \"simulator-w0-6ba087-kserve-5bbd44496f-cmpn2\" (UID: \"2738da06-3225-42c9-b77a-86e864a9babd\") " pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:11:23.161594 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.161525 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2738da06-3225-42c9-b77a-86e864a9babd-tmp-dir\") pod \"simulator-w0-6ba087-kserve-5bbd44496f-cmpn2\" (UID: \"2738da06-3225-42c9-b77a-86e864a9babd\") " pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:11:23.161766 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.161648 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2738da06-3225-42c9-b77a-86e864a9babd-dshm\") pod \"simulator-w0-6ba087-kserve-5bbd44496f-cmpn2\" (UID: \"2738da06-3225-42c9-b77a-86e864a9babd\") " pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:11:23.161766 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.161680 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2738da06-3225-42c9-b77a-86e864a9babd-tls-certs\") pod \"simulator-w0-6ba087-kserve-5bbd44496f-cmpn2\" (UID: \"2738da06-3225-42c9-b77a-86e864a9babd\") " pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:11:23.161902 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.161882 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2738da06-3225-42c9-b77a-86e864a9babd-tmp-dir\") pod \"simulator-w0-6ba087-kserve-5bbd44496f-cmpn2\" (UID: \"2738da06-3225-42c9-b77a-86e864a9babd\") " pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:11:23.161938 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.161922 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2738da06-3225-42c9-b77a-86e864a9babd-home\") pod \"simulator-w0-6ba087-kserve-5bbd44496f-cmpn2\" (UID: \"2738da06-3225-42c9-b77a-86e864a9babd\") " pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:11:23.161977 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.161964 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2738da06-3225-42c9-b77a-86e864a9babd-model-cache\") pod \"simulator-w0-6ba087-kserve-5bbd44496f-cmpn2\" (UID: \"2738da06-3225-42c9-b77a-86e864a9babd\") " pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:11:23.163812 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.163779 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2738da06-3225-42c9-b77a-86e864a9babd-dshm\") pod \"simulator-w0-6ba087-kserve-5bbd44496f-cmpn2\" (UID: \"2738da06-3225-42c9-b77a-86e864a9babd\") " pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:11:23.164166 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.164149 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2738da06-3225-42c9-b77a-86e864a9babd-tls-certs\") pod \"simulator-w0-6ba087-kserve-5bbd44496f-cmpn2\" (UID: \"2738da06-3225-42c9-b77a-86e864a9babd\") " pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:11:23.168969 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.168951 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmvt4\" (UniqueName: \"kubernetes.io/projected/2738da06-3225-42c9-b77a-86e864a9babd-kube-api-access-gmvt4\") pod \"simulator-w0-6ba087-kserve-5bbd44496f-cmpn2\" (UID: \"2738da06-3225-42c9-b77a-86e864a9babd\") " pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:11:23.313787 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.313726 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:11:23.585180 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:23.585152 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2"] Sep 11 01:11:23.587084 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:11:23.587051 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2738da06_3225_42c9_b77a_86e864a9babd.slice/crio-0f7cf5bbb8181d7ed1cadfab4ca1774958d894617e7f212008a482f149d5e53d WatchSource:0}: Error finding container 0f7cf5bbb8181d7ed1cadfab4ca1774958d894617e7f212008a482f149d5e53d: Status 404 returned error can't find the container with id 0f7cf5bbb8181d7ed1cadfab4ca1774958d894617e7f212008a482f149d5e53d Sep 11 01:11:24.284398 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:24.284367 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" event={"ID":"2738da06-3225-42c9-b77a-86e864a9babd","Type":"ContainerStarted","Data":"6575f777023a500f0dcae8fa7dff8a2f859df666f5e1a246d44483ab57e81479"} Sep 11 01:11:24.284398 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:24.284408 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" event={"ID":"2738da06-3225-42c9-b77a-86e864a9babd","Type":"ContainerStarted","Data":"0f7cf5bbb8181d7ed1cadfab4ca1774958d894617e7f212008a482f149d5e53d"} Sep 11 01:11:24.300633 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:24.300587 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" podStartSLOduration=2.300572495 podStartE2EDuration="2.300572495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:11:24.299486653 +0000 UTC m=+811.122940164" watchObservedRunningTime="2026-09-11 01:11:24.300572495 +0000 UTC m=+811.124026040" Sep 11 01:11:27.730053 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:27.729969 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx"] Sep 11 01:11:27.730502 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:27.730220 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" podUID="90d99681-7ef6-4f8e-99dd-9c37e7db47eb" containerName="maas-api" containerID="cri-o://72026c8146ca4d900cd034b7b043dc30b618439301ffc3d267fddb6d2853ff8d" gracePeriod=30 Sep 11 01:11:27.759494 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:27.759459 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:11:27.862779 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:27.862744 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:11:27.962913 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:27.962863 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:11:28.003076 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.003047 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" Sep 11 01:11:28.033342 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.033304 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:11:28.100809 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.100777 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2wnm\" (UniqueName: \"kubernetes.io/projected/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-kube-api-access-f2wnm\") pod \"90d99681-7ef6-4f8e-99dd-9c37e7db47eb\" (UID: \"90d99681-7ef6-4f8e-99dd-9c37e7db47eb\") " Sep 11 01:11:28.101002 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.100943 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-metrics-certs\") pod \"90d99681-7ef6-4f8e-99dd-9c37e7db47eb\" (UID: \"90d99681-7ef6-4f8e-99dd-9c37e7db47eb\") " Sep 11 01:11:28.101002 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.100980 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-openshift-service-ca\") pod \"90d99681-7ef6-4f8e-99dd-9c37e7db47eb\" (UID: \"90d99681-7ef6-4f8e-99dd-9c37e7db47eb\") " Sep 11 01:11:28.101084 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.101015 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-maas-api-tls\") pod \"90d99681-7ef6-4f8e-99dd-9c37e7db47eb\" (UID: \"90d99681-7ef6-4f8e-99dd-9c37e7db47eb\") " Sep 11 01:11:28.101601 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.101558 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "90d99681-7ef6-4f8e-99dd-9c37e7db47eb" (UID: "90d99681-7ef6-4f8e-99dd-9c37e7db47eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:11:28.103668 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.103625 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "90d99681-7ef6-4f8e-99dd-9c37e7db47eb" (UID: "90d99681-7ef6-4f8e-99dd-9c37e7db47eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:11:28.103668 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.103641 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-kube-api-access-f2wnm" (OuterVolumeSpecName: "kube-api-access-f2wnm") pod "90d99681-7ef6-4f8e-99dd-9c37e7db47eb" (UID: "90d99681-7ef6-4f8e-99dd-9c37e7db47eb"). InnerVolumeSpecName "kube-api-access-f2wnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:11:28.103979 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.103957 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "90d99681-7ef6-4f8e-99dd-9c37e7db47eb" (UID: "90d99681-7ef6-4f8e-99dd-9c37e7db47eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:11:28.120356 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.120323 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:11:28.202789 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.202748 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2wnm\" (UniqueName: \"kubernetes.io/projected/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-kube-api-access-f2wnm\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:11:28.202789 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.202786 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-metrics-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:11:28.202789 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.202798 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-openshift-service-ca\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:11:28.203045 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.202807 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90d99681-7ef6-4f8e-99dd-9c37e7db47eb-maas-api-tls\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:11:28.307254 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.307167 2554 generic.go:356] "Generic (PLEG): container finished" podID="90d99681-7ef6-4f8e-99dd-9c37e7db47eb" containerID="72026c8146ca4d900cd034b7b043dc30b618439301ffc3d267fddb6d2853ff8d" exitCode=0 Sep 11 01:11:28.307254 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.307214 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" Sep 11 01:11:28.307448 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.307247 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" event={"ID":"90d99681-7ef6-4f8e-99dd-9c37e7db47eb","Type":"ContainerDied","Data":"72026c8146ca4d900cd034b7b043dc30b618439301ffc3d267fddb6d2853ff8d"} Sep 11 01:11:28.307448 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.307302 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx" event={"ID":"90d99681-7ef6-4f8e-99dd-9c37e7db47eb","Type":"ContainerDied","Data":"68349939366bb947da527c81b32114a59e5fc7f96ef421d9cfccd33b3fe531e4"} Sep 11 01:11:28.307448 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.307327 2554 scope.go:117] "RemoveContainer" containerID="72026c8146ca4d900cd034b7b043dc30b618439301ffc3d267fddb6d2853ff8d" Sep 11 01:11:28.317304 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.317287 2554 scope.go:117] "RemoveContainer" containerID="72026c8146ca4d900cd034b7b043dc30b618439301ffc3d267fddb6d2853ff8d" Sep 11 01:11:28.317554 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:11:28.317534 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72026c8146ca4d900cd034b7b043dc30b618439301ffc3d267fddb6d2853ff8d\": container with ID starting with 72026c8146ca4d900cd034b7b043dc30b618439301ffc3d267fddb6d2853ff8d not found: ID does not exist" containerID="72026c8146ca4d900cd034b7b043dc30b618439301ffc3d267fddb6d2853ff8d" Sep 11 01:11:28.317624 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.317560 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72026c8146ca4d900cd034b7b043dc30b618439301ffc3d267fddb6d2853ff8d"} err="failed to get container status \"72026c8146ca4d900cd034b7b043dc30b618439301ffc3d267fddb6d2853ff8d\": rpc error: code = NotFound desc = could not find container \"72026c8146ca4d900cd034b7b043dc30b618439301ffc3d267fddb6d2853ff8d\": container with ID starting with 72026c8146ca4d900cd034b7b043dc30b618439301ffc3d267fddb6d2853ff8d not found: ID does not exist" Sep 11 01:11:28.328561 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.328539 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx"] Sep 11 01:11:28.334895 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.334873 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1d3c7cc9-6df67fcd88-lv8cx"] Sep 11 01:11:28.581134 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.581053 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:11:28.899247 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:28.899170 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:11:29.626276 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:29.626239 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:11:29.679163 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:29.679134 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90d99681-7ef6-4f8e-99dd-9c37e7db47eb" path="/var/lib/kubelet/pods/90d99681-7ef6-4f8e-99dd-9c37e7db47eb/volumes" Sep 11 01:11:31.347753 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:31.347711 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:11:31.445133 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:31.445083 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:11:32.453373 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:32.453338 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll"] Sep 11 01:11:32.453867 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:32.453821 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90d99681-7ef6-4f8e-99dd-9c37e7db47eb" containerName="maas-api" Sep 11 01:11:32.453867 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:32.453861 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="90d99681-7ef6-4f8e-99dd-9c37e7db47eb" containerName="maas-api" Sep 11 01:11:32.454042 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:32.453970 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="90d99681-7ef6-4f8e-99dd-9c37e7db47eb" containerName="maas-api" Sep 11 01:11:32.458952 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:32.458933 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll" Sep 11 01:11:32.461559 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:32.461528 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-aba078c2-serving-cert\"" Sep 11 01:11:32.466901 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:32.466875 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll"] Sep 11 01:11:32.502564 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:32.502533 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:11:32.545527 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:32.545495 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cb706f-55c8-4098-b18e-98a8531237c8-openshift-service-ca\") pod \"maas-api-e2e-ait-aba078c2-74476d566b-pn7ll\" (UID: \"43cb706f-55c8-4098-b18e-98a8531237c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll" Sep 11 01:11:32.545659 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:32.545531 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43cb706f-55c8-4098-b18e-98a8531237c8-maas-api-tls\") pod \"maas-api-e2e-ait-aba078c2-74476d566b-pn7ll\" (UID: \"43cb706f-55c8-4098-b18e-98a8531237c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll" Sep 11 01:11:32.545659 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:32.545568 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43cb706f-55c8-4098-b18e-98a8531237c8-metrics-certs\") pod \"maas-api-e2e-ait-aba078c2-74476d566b-pn7ll\" (UID: \"43cb706f-55c8-4098-b18e-98a8531237c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll" Sep 11 01:11:32.545659 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:32.545634 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssprm\" (UniqueName: \"kubernetes.io/projected/43cb706f-55c8-4098-b18e-98a8531237c8-kube-api-access-ssprm\") pod \"maas-api-e2e-ait-aba078c2-74476d566b-pn7ll\" (UID: \"43cb706f-55c8-4098-b18e-98a8531237c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll" Sep 11 01:11:32.605948 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:32.605920 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:11:32.646563 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:32.646538 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssprm\" (UniqueName: \"kubernetes.io/projected/43cb706f-55c8-4098-b18e-98a8531237c8-kube-api-access-ssprm\") pod \"maas-api-e2e-ait-aba078c2-74476d566b-pn7ll\" (UID: \"43cb706f-55c8-4098-b18e-98a8531237c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll" Sep 11 01:11:32.646689 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:32.646641 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cb706f-55c8-4098-b18e-98a8531237c8-openshift-service-ca\") pod \"maas-api-e2e-ait-aba078c2-74476d566b-pn7ll\" (UID: \"43cb706f-55c8-4098-b18e-98a8531237c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll" Sep 11 01:11:32.646689 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:32.646670 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43cb706f-55c8-4098-b18e-98a8531237c8-maas-api-tls\") pod \"maas-api-e2e-ait-aba078c2-74476d566b-pn7ll\" (UID: \"43cb706f-55c8-4098-b18e-98a8531237c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll" Sep 11 01:11:32.646805 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:32.646690 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43cb706f-55c8-4098-b18e-98a8531237c8-metrics-certs\") pod \"maas-api-e2e-ait-aba078c2-74476d566b-pn7ll\" (UID: \"43cb706f-55c8-4098-b18e-98a8531237c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll" Sep 11 01:11:32.647295 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:32.647273 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cb706f-55c8-4098-b18e-98a8531237c8-openshift-service-ca\") pod \"maas-api-e2e-ait-aba078c2-74476d566b-pn7ll\" (UID: \"43cb706f-55c8-4098-b18e-98a8531237c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll" Sep 11 01:11:32.649159 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:32.649141 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43cb706f-55c8-4098-b18e-98a8531237c8-maas-api-tls\") pod \"maas-api-e2e-ait-aba078c2-74476d566b-pn7ll\" (UID: \"43cb706f-55c8-4098-b18e-98a8531237c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll" Sep 11 01:11:32.649254 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:32.649184 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43cb706f-55c8-4098-b18e-98a8531237c8-metrics-certs\") pod \"maas-api-e2e-ait-aba078c2-74476d566b-pn7ll\" (UID: \"43cb706f-55c8-4098-b18e-98a8531237c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll" Sep 11 01:11:32.654031 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:32.654009 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssprm\" (UniqueName: \"kubernetes.io/projected/43cb706f-55c8-4098-b18e-98a8531237c8-kube-api-access-ssprm\") pod \"maas-api-e2e-ait-aba078c2-74476d566b-pn7ll\" (UID: \"43cb706f-55c8-4098-b18e-98a8531237c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll" Sep 11 01:11:32.704207 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:32.704151 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:11:32.773263 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:32.773236 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll" Sep 11 01:11:32.965307 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:32.965218 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:11:33.044806 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:33.044729 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:11:33.046316 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:33.046286 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll"] Sep 11 01:11:33.049161 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:11:33.049137 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43cb706f_55c8_4098_b18e_98a8531237c8.slice/crio-e484e85ff90850fbbb2f277c8b7d57c5aa3a7b9612788f71217e365be7031af4 WatchSource:0}: Error finding container e484e85ff90850fbbb2f277c8b7d57c5aa3a7b9612788f71217e365be7031af4: Status 404 returned error can't find the container with id e484e85ff90850fbbb2f277c8b7d57c5aa3a7b9612788f71217e365be7031af4 Sep 11 01:11:33.314172 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:33.314094 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:11:33.314172 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:33.314143 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:11:33.326612 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:33.326590 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:11:33.334797 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:33.334774 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll" event={"ID":"43cb706f-55c8-4098-b18e-98a8531237c8","Type":"ContainerStarted","Data":"e484e85ff90850fbbb2f277c8b7d57c5aa3a7b9612788f71217e365be7031af4"} Sep 11 01:11:33.350476 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:33.350444 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:11:34.341590 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:34.341554 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll" event={"ID":"43cb706f-55c8-4098-b18e-98a8531237c8","Type":"ContainerStarted","Data":"46813af7c7a62f949d72bd3784529ea28b51bd56a38f011ac33a3af3f61f2a5e"} Sep 11 01:11:34.342073 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:34.341653 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll" Sep 11 01:11:34.359272 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:34.359229 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll" podStartSLOduration=1.9954302259999999 podStartE2EDuration="2.359215036s" totalImagesPullingTime="363.78481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:11:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:11:34.357141026 +0000 UTC m=+821.180594536" watchObservedRunningTime="2026-09-11 01:11:34.359215036 +0000 UTC m=+821.182668547" Sep 11 01:11:34.985459 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:34.985426 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:11:40.352233 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:40.352203 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll" Sep 11 01:11:44.131792 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:11:44.131752 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:05.317750 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:05.317701 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:06.400830 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:06.400789 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:12.317771 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:12.317726 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:12.656390 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:12.656294 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:13.184316 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.184277 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll"] Sep 11 01:12:13.184567 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.184519 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll" podUID="43cb706f-55c8-4098-b18e-98a8531237c8" containerName="maas-api" containerID="cri-o://46813af7c7a62f949d72bd3784529ea28b51bd56a38f011ac33a3af3f61f2a5e" gracePeriod=30 Sep 11 01:12:13.213838 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.213797 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:13.280869 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.280803 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:13.382964 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.382925 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:13.447495 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.447436 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll" Sep 11 01:12:13.468040 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.468006 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:13.482346 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.482325 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43cb706f-55c8-4098-b18e-98a8531237c8-metrics-certs\") pod \"43cb706f-55c8-4098-b18e-98a8531237c8\" (UID: \"43cb706f-55c8-4098-b18e-98a8531237c8\") " Sep 11 01:12:13.482484 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.482418 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cb706f-55c8-4098-b18e-98a8531237c8-openshift-service-ca\") pod \"43cb706f-55c8-4098-b18e-98a8531237c8\" (UID: \"43cb706f-55c8-4098-b18e-98a8531237c8\") " Sep 11 01:12:13.482484 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.482440 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssprm\" (UniqueName: \"kubernetes.io/projected/43cb706f-55c8-4098-b18e-98a8531237c8-kube-api-access-ssprm\") pod \"43cb706f-55c8-4098-b18e-98a8531237c8\" (UID: \"43cb706f-55c8-4098-b18e-98a8531237c8\") " Sep 11 01:12:13.482484 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.482460 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43cb706f-55c8-4098-b18e-98a8531237c8-maas-api-tls\") pod \"43cb706f-55c8-4098-b18e-98a8531237c8\" (UID: \"43cb706f-55c8-4098-b18e-98a8531237c8\") " Sep 11 01:12:13.482806 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.482780 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43cb706f-55c8-4098-b18e-98a8531237c8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "43cb706f-55c8-4098-b18e-98a8531237c8" (UID: "43cb706f-55c8-4098-b18e-98a8531237c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:12:13.484561 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.484532 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43cb706f-55c8-4098-b18e-98a8531237c8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "43cb706f-55c8-4098-b18e-98a8531237c8" (UID: "43cb706f-55c8-4098-b18e-98a8531237c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:12:13.484664 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.484560 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43cb706f-55c8-4098-b18e-98a8531237c8-kube-api-access-ssprm" (OuterVolumeSpecName: "kube-api-access-ssprm") pod "43cb706f-55c8-4098-b18e-98a8531237c8" (UID: "43cb706f-55c8-4098-b18e-98a8531237c8"). InnerVolumeSpecName "kube-api-access-ssprm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:12:13.485006 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.484984 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43cb706f-55c8-4098-b18e-98a8531237c8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "43cb706f-55c8-4098-b18e-98a8531237c8" (UID: "43cb706f-55c8-4098-b18e-98a8531237c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:12:13.548630 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.548602 2554 generic.go:356] "Generic (PLEG): container finished" podID="43cb706f-55c8-4098-b18e-98a8531237c8" containerID="46813af7c7a62f949d72bd3784529ea28b51bd56a38f011ac33a3af3f61f2a5e" exitCode=0 Sep 11 01:12:13.548744 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.548651 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll" Sep 11 01:12:13.548744 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.548686 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll" event={"ID":"43cb706f-55c8-4098-b18e-98a8531237c8","Type":"ContainerDied","Data":"46813af7c7a62f949d72bd3784529ea28b51bd56a38f011ac33a3af3f61f2a5e"} Sep 11 01:12:13.548865 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.548758 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll" event={"ID":"43cb706f-55c8-4098-b18e-98a8531237c8","Type":"ContainerDied","Data":"e484e85ff90850fbbb2f277c8b7d57c5aa3a7b9612788f71217e365be7031af4"} Sep 11 01:12:13.548865 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.548794 2554 scope.go:117] "RemoveContainer" containerID="46813af7c7a62f949d72bd3784529ea28b51bd56a38f011ac33a3af3f61f2a5e" Sep 11 01:12:13.558586 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.558472 2554 scope.go:117] "RemoveContainer" containerID="46813af7c7a62f949d72bd3784529ea28b51bd56a38f011ac33a3af3f61f2a5e" Sep 11 01:12:13.558824 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:12:13.558797 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46813af7c7a62f949d72bd3784529ea28b51bd56a38f011ac33a3af3f61f2a5e\": container with ID starting with 46813af7c7a62f949d72bd3784529ea28b51bd56a38f011ac33a3af3f61f2a5e not found: ID does not exist" containerID="46813af7c7a62f949d72bd3784529ea28b51bd56a38f011ac33a3af3f61f2a5e" Sep 11 01:12:13.558916 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.558836 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46813af7c7a62f949d72bd3784529ea28b51bd56a38f011ac33a3af3f61f2a5e"} err="failed to get container status \"46813af7c7a62f949d72bd3784529ea28b51bd56a38f011ac33a3af3f61f2a5e\": rpc error: code = NotFound desc = could not find container \"46813af7c7a62f949d72bd3784529ea28b51bd56a38f011ac33a3af3f61f2a5e\": container with ID starting with 46813af7c7a62f949d72bd3784529ea28b51bd56a38f011ac33a3af3f61f2a5e not found: ID does not exist" Sep 11 01:12:13.583717 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.583695 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43cb706f-55c8-4098-b18e-98a8531237c8-metrics-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:12:13.583795 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.583723 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cb706f-55c8-4098-b18e-98a8531237c8-openshift-service-ca\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:12:13.583795 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.583736 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssprm\" (UniqueName: \"kubernetes.io/projected/43cb706f-55c8-4098-b18e-98a8531237c8-kube-api-access-ssprm\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:12:13.583795 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.583748 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43cb706f-55c8-4098-b18e-98a8531237c8-maas-api-tls\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:12:13.585986 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.585965 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll"] Sep 11 01:12:13.596189 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.596168 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-aba078c2-74476d566b-pn7ll"] Sep 11 01:12:13.679142 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:13.679120 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43cb706f-55c8-4098-b18e-98a8531237c8" path="/var/lib/kubelet/pods/43cb706f-55c8-4098-b18e-98a8531237c8/volumes" Sep 11 01:12:14.590790 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:14.590757 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:15.022894 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:15.022860 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:15.363116 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:15.363024 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:15.590536 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:15.590488 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:16.373336 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:16.373301 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:17.003066 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.003037 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5"] Sep 11 01:12:17.003303 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.003276 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5" podUID="897ee5bb-08dc-4d02-b571-f022f8c8de14" containerName="maas-api" containerID="cri-o://1ba816af8343911dd36d46c5f08ce6f3940418740095c6a453a8ff7f931df9e4" gracePeriod=30 Sep 11 01:12:17.035002 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.034975 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:17.136760 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.136730 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:17.235251 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.235213 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:17.260918 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.260869 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5" Sep 11 01:12:17.314184 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.314162 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/897ee5bb-08dc-4d02-b571-f022f8c8de14-metrics-certs\") pod \"897ee5bb-08dc-4d02-b571-f022f8c8de14\" (UID: \"897ee5bb-08dc-4d02-b571-f022f8c8de14\") " Sep 11 01:12:17.314330 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.314196 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkvzs\" (UniqueName: \"kubernetes.io/projected/897ee5bb-08dc-4d02-b571-f022f8c8de14-kube-api-access-zkvzs\") pod \"897ee5bb-08dc-4d02-b571-f022f8c8de14\" (UID: \"897ee5bb-08dc-4d02-b571-f022f8c8de14\") " Sep 11 01:12:17.314330 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.314270 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/897ee5bb-08dc-4d02-b571-f022f8c8de14-openshift-service-ca\") pod \"897ee5bb-08dc-4d02-b571-f022f8c8de14\" (UID: \"897ee5bb-08dc-4d02-b571-f022f8c8de14\") " Sep 11 01:12:17.314330 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.314308 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/897ee5bb-08dc-4d02-b571-f022f8c8de14-maas-api-tls\") pod \"897ee5bb-08dc-4d02-b571-f022f8c8de14\" (UID: \"897ee5bb-08dc-4d02-b571-f022f8c8de14\") " Sep 11 01:12:17.314657 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.314630 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/897ee5bb-08dc-4d02-b571-f022f8c8de14-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "897ee5bb-08dc-4d02-b571-f022f8c8de14" (UID: "897ee5bb-08dc-4d02-b571-f022f8c8de14"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:12:17.316384 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.316352 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/897ee5bb-08dc-4d02-b571-f022f8c8de14-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "897ee5bb-08dc-4d02-b571-f022f8c8de14" (UID: "897ee5bb-08dc-4d02-b571-f022f8c8de14"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:12:17.316472 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.316390 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/897ee5bb-08dc-4d02-b571-f022f8c8de14-kube-api-access-zkvzs" (OuterVolumeSpecName: "kube-api-access-zkvzs") pod "897ee5bb-08dc-4d02-b571-f022f8c8de14" (UID: "897ee5bb-08dc-4d02-b571-f022f8c8de14"). InnerVolumeSpecName "kube-api-access-zkvzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:12:17.316472 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.316434 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/897ee5bb-08dc-4d02-b571-f022f8c8de14-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "897ee5bb-08dc-4d02-b571-f022f8c8de14" (UID: "897ee5bb-08dc-4d02-b571-f022f8c8de14"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:12:17.329209 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.329170 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:17.415204 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.415179 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/897ee5bb-08dc-4d02-b571-f022f8c8de14-openshift-service-ca\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:12:17.415204 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.415201 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/897ee5bb-08dc-4d02-b571-f022f8c8de14-maas-api-tls\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:12:17.415540 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.415212 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/897ee5bb-08dc-4d02-b571-f022f8c8de14-metrics-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:12:17.415540 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.415220 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkvzs\" (UniqueName: \"kubernetes.io/projected/897ee5bb-08dc-4d02-b571-f022f8c8de14-kube-api-access-zkvzs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:12:17.572167 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.572101 2554 generic.go:356] "Generic (PLEG): container finished" podID="897ee5bb-08dc-4d02-b571-f022f8c8de14" containerID="1ba816af8343911dd36d46c5f08ce6f3940418740095c6a453a8ff7f931df9e4" exitCode=0 Sep 11 01:12:17.572167 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.572153 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5" Sep 11 01:12:17.572335 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.572182 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5" event={"ID":"897ee5bb-08dc-4d02-b571-f022f8c8de14","Type":"ContainerDied","Data":"1ba816af8343911dd36d46c5f08ce6f3940418740095c6a453a8ff7f931df9e4"} Sep 11 01:12:17.572335 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.572233 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5" event={"ID":"897ee5bb-08dc-4d02-b571-f022f8c8de14","Type":"ContainerDied","Data":"9f3fe5786f8ef7b3c3f18626a0abd368dde07cbd943c5f12e5f5a6dca313e3fc"} Sep 11 01:12:17.572335 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.572258 2554 scope.go:117] "RemoveContainer" containerID="1ba816af8343911dd36d46c5f08ce6f3940418740095c6a453a8ff7f931df9e4" Sep 11 01:12:17.581427 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.581411 2554 scope.go:117] "RemoveContainer" containerID="1ba816af8343911dd36d46c5f08ce6f3940418740095c6a453a8ff7f931df9e4" Sep 11 01:12:17.581696 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:12:17.581676 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba816af8343911dd36d46c5f08ce6f3940418740095c6a453a8ff7f931df9e4\": container with ID starting with 1ba816af8343911dd36d46c5f08ce6f3940418740095c6a453a8ff7f931df9e4 not found: ID does not exist" containerID="1ba816af8343911dd36d46c5f08ce6f3940418740095c6a453a8ff7f931df9e4" Sep 11 01:12:17.581750 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.581704 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba816af8343911dd36d46c5f08ce6f3940418740095c6a453a8ff7f931df9e4"} err="failed to get container status \"1ba816af8343911dd36d46c5f08ce6f3940418740095c6a453a8ff7f931df9e4\": rpc error: code = NotFound desc = could not find container \"1ba816af8343911dd36d46c5f08ce6f3940418740095c6a453a8ff7f931df9e4\": container with ID starting with 1ba816af8343911dd36d46c5f08ce6f3940418740095c6a453a8ff7f931df9e4 not found: ID does not exist" Sep 11 01:12:17.610303 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.610284 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5"] Sep 11 01:12:17.618341 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.618322 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dc7dcd-84c78fdb44-wj8b5"] Sep 11 01:12:17.666438 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.666393 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:17.678563 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.678540 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="897ee5bb-08dc-4d02-b571-f022f8c8de14" path="/var/lib/kubelet/pods/897ee5bb-08dc-4d02-b571-f022f8c8de14/volumes" Sep 11 01:12:17.770715 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:17.770681 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:18.814630 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:18.814600 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq"] Sep 11 01:12:18.815267 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:18.815243 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43cb706f-55c8-4098-b18e-98a8531237c8" containerName="maas-api" Sep 11 01:12:18.815267 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:18.815268 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="43cb706f-55c8-4098-b18e-98a8531237c8" containerName="maas-api" Sep 11 01:12:18.815427 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:18.815301 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="897ee5bb-08dc-4d02-b571-f022f8c8de14" containerName="maas-api" Sep 11 01:12:18.815427 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:18.815309 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="897ee5bb-08dc-4d02-b571-f022f8c8de14" containerName="maas-api" Sep 11 01:12:18.815532 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:18.815434 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="43cb706f-55c8-4098-b18e-98a8531237c8" containerName="maas-api" Sep 11 01:12:18.815532 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:18.815453 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="897ee5bb-08dc-4d02-b571-f022f8c8de14" containerName="maas-api" Sep 11 01:12:18.818943 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:18.818921 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" Sep 11 01:12:18.821146 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:18.821121 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-39ea7c6f-serving-cert\"" Sep 11 01:12:18.831027 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:18.831002 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq"] Sep 11 01:12:18.852070 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:18.852020 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:18.926104 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:18.926080 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:18.927150 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:18.927131 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs9vp\" (UniqueName: \"kubernetes.io/projected/a43e520a-c953-40cd-bfec-5d7042605e1a-kube-api-access-rs9vp\") pod \"maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq\" (UID: \"a43e520a-c953-40cd-bfec-5d7042605e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" Sep 11 01:12:18.927242 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:18.927167 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43e520a-c953-40cd-bfec-5d7042605e1a-openshift-service-ca\") pod \"maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq\" (UID: \"a43e520a-c953-40cd-bfec-5d7042605e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" Sep 11 01:12:18.927300 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:18.927232 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a43e520a-c953-40cd-bfec-5d7042605e1a-maas-api-tls\") pod \"maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq\" (UID: \"a43e520a-c953-40cd-bfec-5d7042605e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" Sep 11 01:12:18.927300 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:18.927290 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a43e520a-c953-40cd-bfec-5d7042605e1a-metrics-certs\") pod \"maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq\" (UID: \"a43e520a-c953-40cd-bfec-5d7042605e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" Sep 11 01:12:19.028478 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:19.028442 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs9vp\" (UniqueName: \"kubernetes.io/projected/a43e520a-c953-40cd-bfec-5d7042605e1a-kube-api-access-rs9vp\") pod \"maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq\" (UID: \"a43e520a-c953-40cd-bfec-5d7042605e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" Sep 11 01:12:19.028591 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:19.028495 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43e520a-c953-40cd-bfec-5d7042605e1a-openshift-service-ca\") pod \"maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq\" (UID: \"a43e520a-c953-40cd-bfec-5d7042605e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" Sep 11 01:12:19.028591 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:19.028534 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a43e520a-c953-40cd-bfec-5d7042605e1a-maas-api-tls\") pod \"maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq\" (UID: \"a43e520a-c953-40cd-bfec-5d7042605e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" Sep 11 01:12:19.028591 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:19.028572 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a43e520a-c953-40cd-bfec-5d7042605e1a-metrics-certs\") pod \"maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq\" (UID: \"a43e520a-c953-40cd-bfec-5d7042605e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" Sep 11 01:12:19.028751 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:12:19.028662 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-serving-cert: secret "maas-api-e2e-derive-39ea7c6f-serving-cert" not found Sep 11 01:12:19.028751 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:12:19.028716 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a43e520a-c953-40cd-bfec-5d7042605e1a-maas-api-tls podName:a43e520a-c953-40cd-bfec-5d7042605e1a nodeName:}" failed. No retries permitted until 2026-09-11 01:12:19.528702899 +0000 UTC m=+866.352156391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a43e520a-c953-40cd-bfec-5d7042605e1a-maas-api-tls") pod "maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" (UID: "a43e520a-c953-40cd-bfec-5d7042605e1a") : secret "maas-api-e2e-derive-39ea7c6f-serving-cert" not found Sep 11 01:12:19.029295 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:19.029270 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43e520a-c953-40cd-bfec-5d7042605e1a-openshift-service-ca\") pod \"maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq\" (UID: \"a43e520a-c953-40cd-bfec-5d7042605e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" Sep 11 01:12:19.031125 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:19.031103 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a43e520a-c953-40cd-bfec-5d7042605e1a-metrics-certs\") pod \"maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq\" (UID: \"a43e520a-c953-40cd-bfec-5d7042605e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" Sep 11 01:12:19.037045 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:19.037027 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs9vp\" (UniqueName: \"kubernetes.io/projected/a43e520a-c953-40cd-bfec-5d7042605e1a-kube-api-access-rs9vp\") pod \"maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq\" (UID: \"a43e520a-c953-40cd-bfec-5d7042605e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" Sep 11 01:12:19.258135 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:19.258108 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:19.427913 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:19.427886 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:19.532832 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:19.532763 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a43e520a-c953-40cd-bfec-5d7042605e1a-maas-api-tls\") pod \"maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq\" (UID: \"a43e520a-c953-40cd-bfec-5d7042605e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" Sep 11 01:12:19.533691 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:19.533659 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:19.535201 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:19.535176 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a43e520a-c953-40cd-bfec-5d7042605e1a-maas-api-tls\") pod \"maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq\" (UID: \"a43e520a-c953-40cd-bfec-5d7042605e1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" Sep 11 01:12:19.731711 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:19.731679 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" Sep 11 01:12:20.014221 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:20.014196 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq"] Sep 11 01:12:20.016258 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:12:20.016232 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43e520a_c953_40cd_bfec_5d7042605e1a.slice/crio-35bac8bc0e83bf683cd8d27c82a1002520c09ddd72c83493fca56b5b9c08ed51 WatchSource:0}: Error finding container 35bac8bc0e83bf683cd8d27c82a1002520c09ddd72c83493fca56b5b9c08ed51: Status 404 returned error can't find the container with id 35bac8bc0e83bf683cd8d27c82a1002520c09ddd72c83493fca56b5b9c08ed51 Sep 11 01:12:20.589435 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:20.589404 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" event={"ID":"a43e520a-c953-40cd-bfec-5d7042605e1a","Type":"ContainerStarted","Data":"54846cd21ced4d41e0716d9c4666ea479f4dd66293fe501fe1df734798fae951"} Sep 11 01:12:20.589575 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:20.589459 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" event={"ID":"a43e520a-c953-40cd-bfec-5d7042605e1a","Type":"ContainerStarted","Data":"35bac8bc0e83bf683cd8d27c82a1002520c09ddd72c83493fca56b5b9c08ed51"} Sep 11 01:12:20.589575 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:20.589503 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" Sep 11 01:12:20.608608 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:20.608563 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" podStartSLOduration=2.15581252 podStartE2EDuration="2.608547598s" totalImagesPullingTime="452.735078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:12:20.606104133 +0000 UTC m=+867.429557644" watchObservedRunningTime="2026-09-11 01:12:20.608547598 +0000 UTC m=+867.432001108" Sep 11 01:12:22.707607 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:22.707578 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:26.600302 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:26.600274 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" Sep 11 01:12:27.669907 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:27.669877 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:29.186778 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:29.186741 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:29.287052 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:29.287016 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:29.520334 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:29.520251 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:29.945670 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:29.945627 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:30.247985 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:30.247889 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:30.382750 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:30.382709 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:31.273025 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:31.272968 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:33.332368 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:33.332334 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:33.424675 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:33.424639 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:34.447489 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:34.447451 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg"] Sep 11 01:12:34.452466 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:34.452435 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" Sep 11 01:12:34.455186 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:34.455163 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-8d451c-serving-cert\"" Sep 11 01:12:34.462306 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:34.462281 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg"] Sep 11 01:12:34.478600 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:34.478571 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:34.538955 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:34.538934 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:34.566172 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:34.566149 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63b10eb2-2b57-4a43-abc4-451ffd0e2167-metrics-certs\") pod \"maas-api-e2e-explicit-8d451c-74d9b74947-2gndg\" (UID: \"63b10eb2-2b57-4a43-abc4-451ffd0e2167\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" Sep 11 01:12:34.566278 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:34.566187 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b10eb2-2b57-4a43-abc4-451ffd0e2167-openshift-service-ca\") pod \"maas-api-e2e-explicit-8d451c-74d9b74947-2gndg\" (UID: \"63b10eb2-2b57-4a43-abc4-451ffd0e2167\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" Sep 11 01:12:34.566278 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:34.566251 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63b10eb2-2b57-4a43-abc4-451ffd0e2167-maas-api-tls\") pod \"maas-api-e2e-explicit-8d451c-74d9b74947-2gndg\" (UID: \"63b10eb2-2b57-4a43-abc4-451ffd0e2167\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" Sep 11 01:12:34.566356 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:34.566285 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvl7c\" (UniqueName: \"kubernetes.io/projected/63b10eb2-2b57-4a43-abc4-451ffd0e2167-kube-api-access-mvl7c\") pod \"maas-api-e2e-explicit-8d451c-74d9b74947-2gndg\" (UID: \"63b10eb2-2b57-4a43-abc4-451ffd0e2167\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" Sep 11 01:12:34.666804 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:34.666777 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63b10eb2-2b57-4a43-abc4-451ffd0e2167-metrics-certs\") pod \"maas-api-e2e-explicit-8d451c-74d9b74947-2gndg\" (UID: \"63b10eb2-2b57-4a43-abc4-451ffd0e2167\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" Sep 11 01:12:34.666927 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:34.666813 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b10eb2-2b57-4a43-abc4-451ffd0e2167-openshift-service-ca\") pod \"maas-api-e2e-explicit-8d451c-74d9b74947-2gndg\" (UID: \"63b10eb2-2b57-4a43-abc4-451ffd0e2167\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" Sep 11 01:12:34.666927 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:34.666869 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63b10eb2-2b57-4a43-abc4-451ffd0e2167-maas-api-tls\") pod \"maas-api-e2e-explicit-8d451c-74d9b74947-2gndg\" (UID: \"63b10eb2-2b57-4a43-abc4-451ffd0e2167\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" Sep 11 01:12:34.666927 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:34.666889 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvl7c\" (UniqueName: \"kubernetes.io/projected/63b10eb2-2b57-4a43-abc4-451ffd0e2167-kube-api-access-mvl7c\") pod \"maas-api-e2e-explicit-8d451c-74d9b74947-2gndg\" (UID: \"63b10eb2-2b57-4a43-abc4-451ffd0e2167\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" Sep 11 01:12:34.667052 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:12:34.667009 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-serving-cert: secret "maas-api-e2e-explicit-8d451c-serving-cert" not found Sep 11 01:12:34.667092 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:12:34.667083 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63b10eb2-2b57-4a43-abc4-451ffd0e2167-maas-api-tls podName:63b10eb2-2b57-4a43-abc4-451ffd0e2167 nodeName:}" failed. No retries permitted until 2026-09-11 01:12:35.167064206 +0000 UTC m=+881.990517707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/63b10eb2-2b57-4a43-abc4-451ffd0e2167-maas-api-tls") pod "maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" (UID: "63b10eb2-2b57-4a43-abc4-451ffd0e2167") : secret "maas-api-e2e-explicit-8d451c-serving-cert" not found Sep 11 01:12:34.667569 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:34.667550 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b10eb2-2b57-4a43-abc4-451ffd0e2167-openshift-service-ca\") pod \"maas-api-e2e-explicit-8d451c-74d9b74947-2gndg\" (UID: \"63b10eb2-2b57-4a43-abc4-451ffd0e2167\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" Sep 11 01:12:34.669326 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:34.669297 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63b10eb2-2b57-4a43-abc4-451ffd0e2167-metrics-certs\") pod \"maas-api-e2e-explicit-8d451c-74d9b74947-2gndg\" (UID: \"63b10eb2-2b57-4a43-abc4-451ffd0e2167\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" Sep 11 01:12:34.679577 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:34.679554 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvl7c\" (UniqueName: \"kubernetes.io/projected/63b10eb2-2b57-4a43-abc4-451ffd0e2167-kube-api-access-mvl7c\") pod \"maas-api-e2e-explicit-8d451c-74d9b74947-2gndg\" (UID: \"63b10eb2-2b57-4a43-abc4-451ffd0e2167\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" Sep 11 01:12:34.902083 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:34.902005 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:35.000301 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:35.000269 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:35.080782 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:35.080749 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:35.170647 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:35.170625 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63b10eb2-2b57-4a43-abc4-451ffd0e2167-maas-api-tls\") pod \"maas-api-e2e-explicit-8d451c-74d9b74947-2gndg\" (UID: \"63b10eb2-2b57-4a43-abc4-451ffd0e2167\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" Sep 11 01:12:35.172973 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:35.172947 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63b10eb2-2b57-4a43-abc4-451ffd0e2167-maas-api-tls\") pod \"maas-api-e2e-explicit-8d451c-74d9b74947-2gndg\" (UID: \"63b10eb2-2b57-4a43-abc4-451ffd0e2167\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" Sep 11 01:12:35.364812 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:35.364780 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" Sep 11 01:12:35.636803 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:35.636779 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg"] Sep 11 01:12:35.638945 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:12:35.638918 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b10eb2_2b57_4a43_abc4_451ffd0e2167.slice/crio-0708d66541d51237cbe63eccaded659cb1973b2f7d63192ca55fd3d6c27c379a WatchSource:0}: Error finding container 0708d66541d51237cbe63eccaded659cb1973b2f7d63192ca55fd3d6c27c379a: Status 404 returned error can't find the container with id 0708d66541d51237cbe63eccaded659cb1973b2f7d63192ca55fd3d6c27c379a Sep 11 01:12:35.668955 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:35.668918 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" event={"ID":"63b10eb2-2b57-4a43-abc4-451ffd0e2167","Type":"ContainerStarted","Data":"0708d66541d51237cbe63eccaded659cb1973b2f7d63192ca55fd3d6c27c379a"} Sep 11 01:12:36.675774 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:36.675745 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" event={"ID":"63b10eb2-2b57-4a43-abc4-451ffd0e2167","Type":"ContainerStarted","Data":"8040208392b439610a89d8cda096e1dfb6d518321bbeef81e2aac08e6035747a"} Sep 11 01:12:36.676148 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:36.675888 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" Sep 11 01:12:36.693194 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:36.693153 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" podStartSLOduration=2.354179628 podStartE2EDuration="2.693139778s" totalImagesPullingTime="338.96015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:12:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:12:36.691389863 +0000 UTC m=+883.514843373" watchObservedRunningTime="2026-09-11 01:12:36.693139778 +0000 UTC m=+883.516593288" Sep 11 01:12:39.831400 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:39.831352 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:42.685584 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:42.685556 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" Sep 11 01:12:49.469226 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:49.469154 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq"] Sep 11 01:12:49.469721 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:49.469359 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" podUID="a43e520a-c953-40cd-bfec-5d7042605e1a" containerName="maas-api" containerID="cri-o://54846cd21ced4d41e0716d9c4666ea479f4dd66293fe501fe1df734798fae951" gracePeriod=30 Sep 11 01:12:49.506265 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:49.506237 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:49.627709 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:49.627568 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:49.725965 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:49.725868 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:49.744359 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:49.744332 2554 generic.go:356] "Generic (PLEG): container finished" podID="a43e520a-c953-40cd-bfec-5d7042605e1a" containerID="54846cd21ced4d41e0716d9c4666ea479f4dd66293fe501fe1df734798fae951" exitCode=0 Sep 11 01:12:49.744482 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:49.744403 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" event={"ID":"a43e520a-c953-40cd-bfec-5d7042605e1a","Type":"ContainerDied","Data":"54846cd21ced4d41e0716d9c4666ea479f4dd66293fe501fe1df734798fae951"} Sep 11 01:12:49.744482 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:49.744464 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" event={"ID":"a43e520a-c953-40cd-bfec-5d7042605e1a","Type":"ContainerDied","Data":"35bac8bc0e83bf683cd8d27c82a1002520c09ddd72c83493fca56b5b9c08ed51"} Sep 11 01:12:49.744575 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:49.744488 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35bac8bc0e83bf683cd8d27c82a1002520c09ddd72c83493fca56b5b9c08ed51" Sep 11 01:12:49.758154 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:49.758133 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" Sep 11 01:12:49.828444 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:49.828414 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:49.894001 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:49.893975 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a43e520a-c953-40cd-bfec-5d7042605e1a-maas-api-tls\") pod \"a43e520a-c953-40cd-bfec-5d7042605e1a\" (UID: \"a43e520a-c953-40cd-bfec-5d7042605e1a\") " Sep 11 01:12:49.894147 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:49.894022 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a43e520a-c953-40cd-bfec-5d7042605e1a-metrics-certs\") pod \"a43e520a-c953-40cd-bfec-5d7042605e1a\" (UID: \"a43e520a-c953-40cd-bfec-5d7042605e1a\") " Sep 11 01:12:49.894147 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:49.894084 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43e520a-c953-40cd-bfec-5d7042605e1a-openshift-service-ca\") pod \"a43e520a-c953-40cd-bfec-5d7042605e1a\" (UID: \"a43e520a-c953-40cd-bfec-5d7042605e1a\") " Sep 11 01:12:49.894147 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:49.894134 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rs9vp\" (UniqueName: \"kubernetes.io/projected/a43e520a-c953-40cd-bfec-5d7042605e1a-kube-api-access-rs9vp\") pod \"a43e520a-c953-40cd-bfec-5d7042605e1a\" (UID: \"a43e520a-c953-40cd-bfec-5d7042605e1a\") " Sep 11 01:12:49.894541 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:49.894478 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a43e520a-c953-40cd-bfec-5d7042605e1a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a43e520a-c953-40cd-bfec-5d7042605e1a" (UID: "a43e520a-c953-40cd-bfec-5d7042605e1a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:12:49.897511 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:49.897483 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a43e520a-c953-40cd-bfec-5d7042605e1a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a43e520a-c953-40cd-bfec-5d7042605e1a" (UID: "a43e520a-c953-40cd-bfec-5d7042605e1a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:12:49.898853 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:49.898816 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a43e520a-c953-40cd-bfec-5d7042605e1a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a43e520a-c953-40cd-bfec-5d7042605e1a" (UID: "a43e520a-c953-40cd-bfec-5d7042605e1a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:12:49.898954 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:49.898904 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a43e520a-c953-40cd-bfec-5d7042605e1a-kube-api-access-rs9vp" (OuterVolumeSpecName: "kube-api-access-rs9vp") pod "a43e520a-c953-40cd-bfec-5d7042605e1a" (UID: "a43e520a-c953-40cd-bfec-5d7042605e1a"). InnerVolumeSpecName "kube-api-access-rs9vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:12:49.923073 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:49.923043 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:49.995456 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:49.995398 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a43e520a-c953-40cd-bfec-5d7042605e1a-maas-api-tls\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:12:49.995456 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:49.995421 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a43e520a-c953-40cd-bfec-5d7042605e1a-metrics-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:12:49.995456 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:49.995430 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43e520a-c953-40cd-bfec-5d7042605e1a-openshift-service-ca\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:12:49.995456 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:49.995440 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs9vp\" (UniqueName: \"kubernetes.io/projected/a43e520a-c953-40cd-bfec-5d7042605e1a-kube-api-access-rs9vp\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:12:50.686460 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:50.686424 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:50.757068 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:50.756972 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq" Sep 11 01:12:50.792528 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:50.792487 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:50.792916 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:50.792893 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq"] Sep 11 01:12:50.795654 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:50.795628 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-39ea7c6f-7856dd75df-85cnq"] Sep 11 01:12:51.678726 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:51.678694 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a43e520a-c953-40cd-bfec-5d7042605e1a" path="/var/lib/kubelet/pods/a43e520a-c953-40cd-bfec-5d7042605e1a/volumes" Sep 11 01:12:52.261544 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:52.261510 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:53.635264 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:53.635237 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wtd8v_c2b320dd-43e9-435d-8e4f-2bca1f1233ff/ovn-acl-logging/0.log" Sep 11 01:12:53.636006 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:53.635990 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wtd8v_c2b320dd-43e9-435d-8e4f-2bca1f1233ff/ovn-acl-logging/0.log" Sep 11 01:12:53.789363 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:53.789332 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:55.363716 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:55.363673 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:57.977431 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:57.977391 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:58.085456 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:58.085427 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:58.177897 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:58.177865 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:58.277458 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:58.277387 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:58.379134 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:58.379106 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:12:58.689952 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:12:58.689917 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:13:03.819681 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:03.819650 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:13:16.959293 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:16.959251 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:13:24.658521 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:24.658459 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:13:28.807819 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:28.807789 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr"] Sep 11 01:13:28.808187 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:28.808165 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a43e520a-c953-40cd-bfec-5d7042605e1a" containerName="maas-api" Sep 11 01:13:28.808187 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:28.808178 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a43e520a-c953-40cd-bfec-5d7042605e1a" containerName="maas-api" Sep 11 01:13:28.808258 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:28.808254 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a43e520a-c953-40cd-bfec-5d7042605e1a" containerName="maas-api" Sep 11 01:13:28.811526 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:28.811507 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:13:28.814345 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:28.814323 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-891e67\"/\"kube-root-ca.crt\"" Sep 11 01:13:28.815563 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:28.815545 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-891e67\"/\"default-dockercfg-c82lg\"" Sep 11 01:13:28.815661 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:28.815550 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-891e67\"/\"openshift-service-ca.crt\"" Sep 11 01:13:28.815661 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:28.815555 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-891e67\"/\"test-model-891e67-kserve-self-signed-certs\"" Sep 11 01:13:28.821047 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:28.821025 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr"] Sep 11 01:13:28.889866 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:28.889825 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:13:28.924895 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:28.924870 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-home\") pod \"test-model-891e67-kserve-55bfc86ffc-nf9kr\" (UID: \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\") " pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:13:28.925020 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:28.924914 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-model-cache\") pod \"test-model-891e67-kserve-55bfc86ffc-nf9kr\" (UID: \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\") " pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:13:28.925020 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:28.924975 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzh2f\" (UniqueName: \"kubernetes.io/projected/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-kube-api-access-pzh2f\") pod \"test-model-891e67-kserve-55bfc86ffc-nf9kr\" (UID: \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\") " pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:13:28.925020 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:28.924994 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-dshm\") pod \"test-model-891e67-kserve-55bfc86ffc-nf9kr\" (UID: \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\") " pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:13:28.925172 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:28.925040 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-tls-certs\") pod \"test-model-891e67-kserve-55bfc86ffc-nf9kr\" (UID: \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\") " pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:13:28.925172 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:28.925063 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-tmp-dir\") pod \"test-model-891e67-kserve-55bfc86ffc-nf9kr\" (UID: \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\") " pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:13:29.026131 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:29.026105 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-tls-certs\") pod \"test-model-891e67-kserve-55bfc86ffc-nf9kr\" (UID: \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\") " pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:13:29.026260 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:29.026139 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-tmp-dir\") pod \"test-model-891e67-kserve-55bfc86ffc-nf9kr\" (UID: \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\") " pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:13:29.026260 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:29.026165 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-home\") pod \"test-model-891e67-kserve-55bfc86ffc-nf9kr\" (UID: \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\") " pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:13:29.026260 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:29.026197 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-model-cache\") pod \"test-model-891e67-kserve-55bfc86ffc-nf9kr\" (UID: \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\") " pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:13:29.026260 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:29.026252 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzh2f\" (UniqueName: \"kubernetes.io/projected/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-kube-api-access-pzh2f\") pod \"test-model-891e67-kserve-55bfc86ffc-nf9kr\" (UID: \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\") " pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:13:29.026485 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:29.026278 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-dshm\") pod \"test-model-891e67-kserve-55bfc86ffc-nf9kr\" (UID: \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\") " pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:13:29.026590 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:29.026567 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-tmp-dir\") pod \"test-model-891e67-kserve-55bfc86ffc-nf9kr\" (UID: \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\") " pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:13:29.026704 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:29.026674 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-model-cache\") pod \"test-model-891e67-kserve-55bfc86ffc-nf9kr\" (UID: \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\") " pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:13:29.026704 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:29.026700 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-home\") pod \"test-model-891e67-kserve-55bfc86ffc-nf9kr\" (UID: \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\") " pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:13:29.028558 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:29.028530 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-dshm\") pod \"test-model-891e67-kserve-55bfc86ffc-nf9kr\" (UID: \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\") " pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:13:29.028641 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:29.028560 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-tls-certs\") pod \"test-model-891e67-kserve-55bfc86ffc-nf9kr\" (UID: \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\") " pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:13:29.036092 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:29.036068 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzh2f\" (UniqueName: \"kubernetes.io/projected/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-kube-api-access-pzh2f\") pod \"test-model-891e67-kserve-55bfc86ffc-nf9kr\" (UID: \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\") " pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:13:29.121614 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:29.121562 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:13:29.397003 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:29.396979 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr"] Sep 11 01:13:29.398895 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:13:29.398821 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded80a8bf_9214_4bfc_80c8_896c66d5e5ff.slice/crio-dc3cf47de600c5404b9c6309087120a0417e45ba99c9bb72310d1f287e4f62bf WatchSource:0}: Error finding container dc3cf47de600c5404b9c6309087120a0417e45ba99c9bb72310d1f287e4f62bf: Status 404 returned error can't find the container with id dc3cf47de600c5404b9c6309087120a0417e45ba99c9bb72310d1f287e4f62bf Sep 11 01:13:29.960485 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:29.960444 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" event={"ID":"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff","Type":"ContainerStarted","Data":"e7982ca5be61adb9a64096013af2ed84edc6f21fbf201448c08be8eb9934ad38"} Sep 11 01:13:29.960883 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:29.960490 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" event={"ID":"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff","Type":"ContainerStarted","Data":"dc3cf47de600c5404b9c6309087120a0417e45ba99c9bb72310d1f287e4f62bf"} Sep 11 01:13:29.981421 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:29.981377 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" podStartSLOduration=1.981362142 podStartE2EDuration="1.981362142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:13:29.978985862 +0000 UTC m=+936.802439373" watchObservedRunningTime="2026-09-11 01:13:29.981362142 +0000 UTC m=+936.804815652" Sep 11 01:13:30.972326 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:30.972295 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:13:31.035905 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:31.035872 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:13:31.114661 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:31.114624 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:13:31.215421 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:31.215391 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:13:39.121970 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:39.121927 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:13:39.122361 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:39.121982 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:13:39.135284 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:39.135263 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:13:40.021533 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:40.021502 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:13:47.012265 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:47.012219 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:13:51.195459 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.195429 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd"] Sep 11 01:13:51.198771 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.198752 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:13:51.202343 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.202319 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-8d451c\"/\"explicit-model-8d451c-kserve-self-signed-certs\"" Sep 11 01:13:51.202449 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.202328 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-8d451c\"/\"default-dockercfg-vsh7h\"" Sep 11 01:13:51.202602 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.202584 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-8d451c\"/\"openshift-service-ca.crt\"" Sep 11 01:13:51.202693 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.202639 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-8d451c\"/\"kube-root-ca.crt\"" Sep 11 01:13:51.209953 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.209925 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd"] Sep 11 01:13:51.220029 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.219981 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:13:51.317338 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.317308 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-tls-certs\") pod \"explicit-model-8d451c-kserve-6659d64779-lnnfd\" (UID: \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\") " pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:13:51.317470 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.317348 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-tmp-dir\") pod \"explicit-model-8d451c-kserve-6659d64779-lnnfd\" (UID: \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\") " pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:13:51.317470 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.317412 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-model-cache\") pod \"explicit-model-8d451c-kserve-6659d64779-lnnfd\" (UID: \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\") " pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:13:51.317470 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.317452 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz5cq\" (UniqueName: \"kubernetes.io/projected/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-kube-api-access-fz5cq\") pod \"explicit-model-8d451c-kserve-6659d64779-lnnfd\" (UID: \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\") " pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:13:51.317601 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.317514 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-dshm\") pod \"explicit-model-8d451c-kserve-6659d64779-lnnfd\" (UID: \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\") " pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:13:51.317601 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.317542 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-home\") pod \"explicit-model-8d451c-kserve-6659d64779-lnnfd\" (UID: \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\") " pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:13:51.418617 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.418585 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-tmp-dir\") pod \"explicit-model-8d451c-kserve-6659d64779-lnnfd\" (UID: \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\") " pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:13:51.418764 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.418635 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-model-cache\") pod \"explicit-model-8d451c-kserve-6659d64779-lnnfd\" (UID: \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\") " pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:13:51.418764 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.418659 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz5cq\" (UniqueName: \"kubernetes.io/projected/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-kube-api-access-fz5cq\") pod \"explicit-model-8d451c-kserve-6659d64779-lnnfd\" (UID: \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\") " pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:13:51.418764 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.418696 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-dshm\") pod \"explicit-model-8d451c-kserve-6659d64779-lnnfd\" (UID: \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\") " pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:13:51.418968 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.418813 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-home\") pod \"explicit-model-8d451c-kserve-6659d64779-lnnfd\" (UID: \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\") " pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:13:51.418968 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.418891 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-tls-certs\") pod \"explicit-model-8d451c-kserve-6659d64779-lnnfd\" (UID: \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\") " pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:13:51.418968 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.418931 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-tmp-dir\") pod \"explicit-model-8d451c-kserve-6659d64779-lnnfd\" (UID: \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\") " pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:13:51.419116 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.419054 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-model-cache\") pod \"explicit-model-8d451c-kserve-6659d64779-lnnfd\" (UID: \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\") " pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:13:51.419188 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.419168 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-home\") pod \"explicit-model-8d451c-kserve-6659d64779-lnnfd\" (UID: \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\") " pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:13:51.421070 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.421042 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-dshm\") pod \"explicit-model-8d451c-kserve-6659d64779-lnnfd\" (UID: \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\") " pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:13:51.421247 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.421229 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-tls-certs\") pod \"explicit-model-8d451c-kserve-6659d64779-lnnfd\" (UID: \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\") " pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:13:51.427064 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.427046 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz5cq\" (UniqueName: \"kubernetes.io/projected/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-kube-api-access-fz5cq\") pod \"explicit-model-8d451c-kserve-6659d64779-lnnfd\" (UID: \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\") " pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:13:51.519362 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.519291 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:13:51.788473 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:51.788450 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd"] Sep 11 01:13:51.790534 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:13:51.790499 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98974e74_7bfb_4ecd_9f1d_d1cc85d906fc.slice/crio-8316e5c425e14ecc97ec0691a36cdf5515a6611f0807a1bc16ef39a21c4e2910 WatchSource:0}: Error finding container 8316e5c425e14ecc97ec0691a36cdf5515a6611f0807a1bc16ef39a21c4e2910: Status 404 returned error can't find the container with id 8316e5c425e14ecc97ec0691a36cdf5515a6611f0807a1bc16ef39a21c4e2910 Sep 11 01:13:52.073023 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:52.072945 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" event={"ID":"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc","Type":"ContainerStarted","Data":"d4ba51b0eb42a01f6ed76cfe1101ea65c928f6e8d5c1f97185a8cbe9d2db284a"} Sep 11 01:13:52.073023 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:52.072992 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" event={"ID":"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc","Type":"ContainerStarted","Data":"8316e5c425e14ecc97ec0691a36cdf5515a6611f0807a1bc16ef39a21c4e2910"} Sep 11 01:13:52.091027 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:52.090982 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" podStartSLOduration=1.090966023 podStartE2EDuration="1.090966023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:13:52.089068314 +0000 UTC m=+958.912521824" watchObservedRunningTime="2026-09-11 01:13:52.090966023 +0000 UTC m=+958.914419533" Sep 11 01:13:53.095104 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:53.095067 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:13:53.154996 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:53.154964 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:13:53.239341 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:53.239308 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:13:53.339307 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:13:53.339278 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:01.519895 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:01.519836 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:14:01.520360 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:01.519908 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:14:01.531898 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:01.531873 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:14:02.139124 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:02.139097 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:14:03.792383 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:03.792344 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:04.535390 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:04.535354 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:04.635647 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:04.635612 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:10.407238 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:10.407202 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:10.517649 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:10.517617 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:10.734225 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:10.734196 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:10.813470 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:10.813425 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:11.370249 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:11.370213 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:11.469786 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:11.469752 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:11.572342 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:11.572299 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:12.974392 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:12.974360 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:18.801741 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:18.801658 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:18.902163 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:18.902124 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:19.468616 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:19.468579 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:25.350606 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:25.350550 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:25.411530 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:25.411497 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:25.621525 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:25.621450 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:25.877423 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:25.877318 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:26.247569 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:26.247533 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:31.748609 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:31.748570 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:31.839920 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:31.839889 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:32.843904 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:32.843869 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt"] Sep 11 01:14:32.849448 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:32.849426 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" Sep 11 01:14:32.851584 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:32.851556 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7b8e72c7-serving-cert\"" Sep 11 01:14:32.862770 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:32.862746 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt"] Sep 11 01:14:32.883921 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:32.883899 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:32.974148 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:32.974115 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9896ad1-58bb-4e9e-8aab-276ba405853e-maas-api-tls\") pod \"maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt\" (UID: \"f9896ad1-58bb-4e9e-8aab-276ba405853e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" Sep 11 01:14:32.974148 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:32.974152 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9896ad1-58bb-4e9e-8aab-276ba405853e-metrics-certs\") pod \"maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt\" (UID: \"f9896ad1-58bb-4e9e-8aab-276ba405853e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" Sep 11 01:14:32.974316 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:32.974241 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrbmb\" (UniqueName: \"kubernetes.io/projected/f9896ad1-58bb-4e9e-8aab-276ba405853e-kube-api-access-qrbmb\") pod \"maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt\" (UID: \"f9896ad1-58bb-4e9e-8aab-276ba405853e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" Sep 11 01:14:32.974364 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:32.974321 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9896ad1-58bb-4e9e-8aab-276ba405853e-openshift-service-ca\") pod \"maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt\" (UID: \"f9896ad1-58bb-4e9e-8aab-276ba405853e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" Sep 11 01:14:32.984754 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:32.984697 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:33.075785 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:33.075762 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrbmb\" (UniqueName: \"kubernetes.io/projected/f9896ad1-58bb-4e9e-8aab-276ba405853e-kube-api-access-qrbmb\") pod \"maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt\" (UID: \"f9896ad1-58bb-4e9e-8aab-276ba405853e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" Sep 11 01:14:33.075916 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:33.075811 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9896ad1-58bb-4e9e-8aab-276ba405853e-openshift-service-ca\") pod \"maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt\" (UID: \"f9896ad1-58bb-4e9e-8aab-276ba405853e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" Sep 11 01:14:33.075916 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:33.075873 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9896ad1-58bb-4e9e-8aab-276ba405853e-maas-api-tls\") pod \"maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt\" (UID: \"f9896ad1-58bb-4e9e-8aab-276ba405853e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" Sep 11 01:14:33.075916 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:33.075895 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9896ad1-58bb-4e9e-8aab-276ba405853e-metrics-certs\") pod \"maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt\" (UID: \"f9896ad1-58bb-4e9e-8aab-276ba405853e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" Sep 11 01:14:33.076032 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:14:33.076005 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-serving-cert: secret "maas-api-e2e-mt-7b8e72c7-serving-cert" not found Sep 11 01:14:33.076082 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:14:33.076070 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9896ad1-58bb-4e9e-8aab-276ba405853e-maas-api-tls podName:f9896ad1-58bb-4e9e-8aab-276ba405853e nodeName:}" failed. No retries permitted until 2026-09-11 01:14:33.576051147 +0000 UTC m=+1000.399504639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f9896ad1-58bb-4e9e-8aab-276ba405853e-maas-api-tls") pod "maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" (UID: "f9896ad1-58bb-4e9e-8aab-276ba405853e") : secret "maas-api-e2e-mt-7b8e72c7-serving-cert" not found Sep 11 01:14:33.076479 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:33.076459 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9896ad1-58bb-4e9e-8aab-276ba405853e-openshift-service-ca\") pod \"maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt\" (UID: \"f9896ad1-58bb-4e9e-8aab-276ba405853e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" Sep 11 01:14:33.078296 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:33.078277 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9896ad1-58bb-4e9e-8aab-276ba405853e-metrics-certs\") pod \"maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt\" (UID: \"f9896ad1-58bb-4e9e-8aab-276ba405853e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" Sep 11 01:14:33.084382 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:33.084359 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrbmb\" (UniqueName: \"kubernetes.io/projected/f9896ad1-58bb-4e9e-8aab-276ba405853e-kube-api-access-qrbmb\") pod \"maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt\" (UID: \"f9896ad1-58bb-4e9e-8aab-276ba405853e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" Sep 11 01:14:33.167655 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:33.167578 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:33.372217 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:33.372186 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:33.478230 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:33.478189 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:33.579758 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:33.579729 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9896ad1-58bb-4e9e-8aab-276ba405853e-maas-api-tls\") pod \"maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt\" (UID: \"f9896ad1-58bb-4e9e-8aab-276ba405853e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" Sep 11 01:14:33.582077 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:33.582056 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9896ad1-58bb-4e9e-8aab-276ba405853e-maas-api-tls\") pod \"maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt\" (UID: \"f9896ad1-58bb-4e9e-8aab-276ba405853e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" Sep 11 01:14:33.762803 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:33.762718 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" Sep 11 01:14:34.035963 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:34.035936 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt"] Sep 11 01:14:34.037409 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:14:34.037385 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9896ad1_58bb_4e9e_8aab_276ba405853e.slice/crio-dcaacc5c6a639578ccffdfff8f20707b7cb3cd45ecb95c6a6847c4ae534f40af WatchSource:0}: Error finding container dcaacc5c6a639578ccffdfff8f20707b7cb3cd45ecb95c6a6847c4ae534f40af: Status 404 returned error can't find the container with id dcaacc5c6a639578ccffdfff8f20707b7cb3cd45ecb95c6a6847c4ae534f40af Sep 11 01:14:34.038588 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:34.038570 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 01:14:34.293275 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:34.293198 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" event={"ID":"f9896ad1-58bb-4e9e-8aab-276ba405853e","Type":"ContainerStarted","Data":"dcaacc5c6a639578ccffdfff8f20707b7cb3cd45ecb95c6a6847c4ae534f40af"} Sep 11 01:14:35.299517 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:35.299487 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" event={"ID":"f9896ad1-58bb-4e9e-8aab-276ba405853e","Type":"ContainerStarted","Data":"87dab9534814d60e0a375c147c6c7ca79a6e1efdb9eb9e3de1930b98f10fbd55"} Sep 11 01:14:35.299904 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:35.299550 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" Sep 11 01:14:35.317520 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:35.317474 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" podStartSLOduration=2.769460884 podStartE2EDuration="3.31746052s" totalImagesPullingTime="547.999636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:14:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:14:35.314979233 +0000 UTC m=+1002.138432742" watchObservedRunningTime="2026-09-11 01:14:35.31746052 +0000 UTC m=+1002.140914029" Sep 11 01:14:36.356639 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:36.356601 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:36.887562 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:36.887527 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg"] Sep 11 01:14:36.887770 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:36.887747 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" podUID="63b10eb2-2b57-4a43-abc4-451ffd0e2167" containerName="maas-api" containerID="cri-o://8040208392b439610a89d8cda096e1dfb6d518321bbeef81e2aac08e6035747a" gracePeriod=30 Sep 11 01:14:36.913456 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:36.913429 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:37.012402 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.012366 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:37.088782 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.088737 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:37.142128 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.142064 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:37.155631 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.155606 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" Sep 11 01:14:37.310507 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.310475 2554 generic.go:356] "Generic (PLEG): container finished" podID="63b10eb2-2b57-4a43-abc4-451ffd0e2167" containerID="8040208392b439610a89d8cda096e1dfb6d518321bbeef81e2aac08e6035747a" exitCode=0 Sep 11 01:14:37.310668 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.310552 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" Sep 11 01:14:37.310668 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.310595 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" event={"ID":"63b10eb2-2b57-4a43-abc4-451ffd0e2167","Type":"ContainerDied","Data":"8040208392b439610a89d8cda096e1dfb6d518321bbeef81e2aac08e6035747a"} Sep 11 01:14:37.310668 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.310651 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg" event={"ID":"63b10eb2-2b57-4a43-abc4-451ffd0e2167","Type":"ContainerDied","Data":"0708d66541d51237cbe63eccaded659cb1973b2f7d63192ca55fd3d6c27c379a"} Sep 11 01:14:37.310888 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.310677 2554 scope.go:117] "RemoveContainer" containerID="8040208392b439610a89d8cda096e1dfb6d518321bbeef81e2aac08e6035747a" Sep 11 01:14:37.312990 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.312958 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63b10eb2-2b57-4a43-abc4-451ffd0e2167-maas-api-tls\") pod \"63b10eb2-2b57-4a43-abc4-451ffd0e2167\" (UID: \"63b10eb2-2b57-4a43-abc4-451ffd0e2167\") " Sep 11 01:14:37.313104 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.313060 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b10eb2-2b57-4a43-abc4-451ffd0e2167-openshift-service-ca\") pod \"63b10eb2-2b57-4a43-abc4-451ffd0e2167\" (UID: \"63b10eb2-2b57-4a43-abc4-451ffd0e2167\") " Sep 11 01:14:37.313162 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.313117 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63b10eb2-2b57-4a43-abc4-451ffd0e2167-metrics-certs\") pod \"63b10eb2-2b57-4a43-abc4-451ffd0e2167\" (UID: \"63b10eb2-2b57-4a43-abc4-451ffd0e2167\") " Sep 11 01:14:37.313214 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.313182 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvl7c\" (UniqueName: \"kubernetes.io/projected/63b10eb2-2b57-4a43-abc4-451ffd0e2167-kube-api-access-mvl7c\") pod \"63b10eb2-2b57-4a43-abc4-451ffd0e2167\" (UID: \"63b10eb2-2b57-4a43-abc4-451ffd0e2167\") " Sep 11 01:14:37.313582 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.313555 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63b10eb2-2b57-4a43-abc4-451ffd0e2167-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "63b10eb2-2b57-4a43-abc4-451ffd0e2167" (UID: "63b10eb2-2b57-4a43-abc4-451ffd0e2167"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:14:37.316030 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.316005 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63b10eb2-2b57-4a43-abc4-451ffd0e2167-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "63b10eb2-2b57-4a43-abc4-451ffd0e2167" (UID: "63b10eb2-2b57-4a43-abc4-451ffd0e2167"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:14:37.316464 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.316435 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63b10eb2-2b57-4a43-abc4-451ffd0e2167-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "63b10eb2-2b57-4a43-abc4-451ffd0e2167" (UID: "63b10eb2-2b57-4a43-abc4-451ffd0e2167"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:14:37.316569 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.316547 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b10eb2-2b57-4a43-abc4-451ffd0e2167-kube-api-access-mvl7c" (OuterVolumeSpecName: "kube-api-access-mvl7c") pod "63b10eb2-2b57-4a43-abc4-451ffd0e2167" (UID: "63b10eb2-2b57-4a43-abc4-451ffd0e2167"). InnerVolumeSpecName "kube-api-access-mvl7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:14:37.335509 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.335480 2554 scope.go:117] "RemoveContainer" containerID="8040208392b439610a89d8cda096e1dfb6d518321bbeef81e2aac08e6035747a" Sep 11 01:14:37.335800 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:14:37.335777 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8040208392b439610a89d8cda096e1dfb6d518321bbeef81e2aac08e6035747a\": container with ID starting with 8040208392b439610a89d8cda096e1dfb6d518321bbeef81e2aac08e6035747a not found: ID does not exist" containerID="8040208392b439610a89d8cda096e1dfb6d518321bbeef81e2aac08e6035747a" Sep 11 01:14:37.335884 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.335809 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8040208392b439610a89d8cda096e1dfb6d518321bbeef81e2aac08e6035747a"} err="failed to get container status \"8040208392b439610a89d8cda096e1dfb6d518321bbeef81e2aac08e6035747a\": rpc error: code = NotFound desc = could not find container \"8040208392b439610a89d8cda096e1dfb6d518321bbeef81e2aac08e6035747a\": container with ID starting with 8040208392b439610a89d8cda096e1dfb6d518321bbeef81e2aac08e6035747a not found: ID does not exist" Sep 11 01:14:37.414137 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.414078 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63b10eb2-2b57-4a43-abc4-451ffd0e2167-maas-api-tls\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:14:37.414137 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.414100 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b10eb2-2b57-4a43-abc4-451ffd0e2167-openshift-service-ca\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:14:37.414137 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.414110 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63b10eb2-2b57-4a43-abc4-451ffd0e2167-metrics-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:14:37.414137 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.414120 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvl7c\" (UniqueName: \"kubernetes.io/projected/63b10eb2-2b57-4a43-abc4-451ffd0e2167-kube-api-access-mvl7c\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:14:37.437954 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.437929 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:37.631708 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.631677 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg"] Sep 11 01:14:37.635339 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.635310 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8d451c-74d9b74947-2gndg"] Sep 11 01:14:37.679194 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:37.679173 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63b10eb2-2b57-4a43-abc4-451ffd0e2167" path="/var/lib/kubelet/pods/63b10eb2-2b57-4a43-abc4-451ffd0e2167/volumes" Sep 11 01:14:41.309907 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:41.309874 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" Sep 11 01:14:42.420474 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:42.420441 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd"] Sep 11 01:14:42.513603 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:42.513565 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:53.956072 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:53.956036 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:56.678864 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:56.678815 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:57.296416 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:57.296384 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:57.396563 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:57.396528 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:57.439431 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:57.439397 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" podUID="98974e74-7bfb-4ecd-9f1d-d1cc85d906fc" containerName="main" containerID="cri-o://d4ba51b0eb42a01f6ed76cfe1101ea65c928f6e8d5c1f97185a8cbe9d2db284a" gracePeriod=45 Sep 11 01:14:57.693851 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:57.693826 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:14:57.697721 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:57.697699 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-model-cache\") pod \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\" (UID: \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\") " Sep 11 01:14:57.697829 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:57.697734 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz5cq\" (UniqueName: \"kubernetes.io/projected/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-kube-api-access-fz5cq\") pod \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\" (UID: \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\") " Sep 11 01:14:57.697829 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:57.697789 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-tmp-dir\") pod \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\" (UID: \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\") " Sep 11 01:14:57.697829 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:57.697817 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-dshm\") pod \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\" (UID: \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\") " Sep 11 01:14:57.698002 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:57.697837 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-home\") pod \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\" (UID: \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\") " Sep 11 01:14:57.698002 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:57.697890 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-tls-certs\") pod \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\" (UID: \"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc\") " Sep 11 01:14:57.698104 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:57.698003 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "98974e74-7bfb-4ecd-9f1d-d1cc85d906fc" (UID: "98974e74-7bfb-4ecd-9f1d-d1cc85d906fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:14:57.698104 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:57.698083 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "98974e74-7bfb-4ecd-9f1d-d1cc85d906fc" (UID: "98974e74-7bfb-4ecd-9f1d-d1cc85d906fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:14:57.698197 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:57.698161 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-home" (OuterVolumeSpecName: "home") pod "98974e74-7bfb-4ecd-9f1d-d1cc85d906fc" (UID: "98974e74-7bfb-4ecd-9f1d-d1cc85d906fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:14:57.698346 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:57.698326 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-tmp-dir\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:14:57.698413 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:57.698349 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-home\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:14:57.698413 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:57.698361 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-model-cache\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:14:57.700151 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:57.700119 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-dshm" (OuterVolumeSpecName: "dshm") pod "98974e74-7bfb-4ecd-9f1d-d1cc85d906fc" (UID: "98974e74-7bfb-4ecd-9f1d-d1cc85d906fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:14:57.700151 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:57.700132 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-kube-api-access-fz5cq" (OuterVolumeSpecName: "kube-api-access-fz5cq") pod "98974e74-7bfb-4ecd-9f1d-d1cc85d906fc" (UID: "98974e74-7bfb-4ecd-9f1d-d1cc85d906fc"). InnerVolumeSpecName "kube-api-access-fz5cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:14:57.700294 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:57.700174 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "98974e74-7bfb-4ecd-9f1d-d1cc85d906fc" (UID: "98974e74-7bfb-4ecd-9f1d-d1cc85d906fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:14:57.705215 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:57.705193 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:14:57.798881 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:57.798830 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-tls-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:14:57.798881 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:57.798876 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz5cq\" (UniqueName: \"kubernetes.io/projected/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-kube-api-access-fz5cq\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:14:57.798881 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:57.798886 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc-dshm\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:14:58.425166 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:58.425135 2554 generic.go:356] "Generic (PLEG): container finished" podID="98974e74-7bfb-4ecd-9f1d-d1cc85d906fc" containerID="d4ba51b0eb42a01f6ed76cfe1101ea65c928f6e8d5c1f97185a8cbe9d2db284a" exitCode=0 Sep 11 01:14:58.425364 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:58.425192 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" Sep 11 01:14:58.425364 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:58.425196 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" event={"ID":"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc","Type":"ContainerDied","Data":"d4ba51b0eb42a01f6ed76cfe1101ea65c928f6e8d5c1f97185a8cbe9d2db284a"} Sep 11 01:14:58.425364 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:58.425231 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd" event={"ID":"98974e74-7bfb-4ecd-9f1d-d1cc85d906fc","Type":"ContainerDied","Data":"8316e5c425e14ecc97ec0691a36cdf5515a6611f0807a1bc16ef39a21c4e2910"} Sep 11 01:14:58.425364 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:58.425249 2554 scope.go:117] "RemoveContainer" containerID="d4ba51b0eb42a01f6ed76cfe1101ea65c928f6e8d5c1f97185a8cbe9d2db284a" Sep 11 01:14:58.434697 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:58.434681 2554 scope.go:117] "RemoveContainer" containerID="d4ba51b0eb42a01f6ed76cfe1101ea65c928f6e8d5c1f97185a8cbe9d2db284a" Sep 11 01:14:58.434984 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:14:58.434964 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ba51b0eb42a01f6ed76cfe1101ea65c928f6e8d5c1f97185a8cbe9d2db284a\": container with ID starting with d4ba51b0eb42a01f6ed76cfe1101ea65c928f6e8d5c1f97185a8cbe9d2db284a not found: ID does not exist" containerID="d4ba51b0eb42a01f6ed76cfe1101ea65c928f6e8d5c1f97185a8cbe9d2db284a" Sep 11 01:14:58.435053 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:58.434992 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ba51b0eb42a01f6ed76cfe1101ea65c928f6e8d5c1f97185a8cbe9d2db284a"} err="failed to get container status \"d4ba51b0eb42a01f6ed76cfe1101ea65c928f6e8d5c1f97185a8cbe9d2db284a\": rpc error: code = NotFound desc = could not find container \"d4ba51b0eb42a01f6ed76cfe1101ea65c928f6e8d5c1f97185a8cbe9d2db284a\": container with ID starting with d4ba51b0eb42a01f6ed76cfe1101ea65c928f6e8d5c1f97185a8cbe9d2db284a not found: ID does not exist" Sep 11 01:14:58.449093 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:58.449066 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd"] Sep 11 01:14:58.456863 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:58.455863 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-8d451c/explicit-model-8d451c-kserve-6659d64779-lnnfd"] Sep 11 01:14:59.678563 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:14:59.678529 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98974e74-7bfb-4ecd-9f1d-d1cc85d906fc" path="/var/lib/kubelet/pods/98974e74-7bfb-4ecd-9f1d-d1cc85d906fc/volumes" Sep 11 01:15:03.484014 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:03.483981 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:03.543991 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:03.543953 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:10.502447 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.502412 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r"] Sep 11 01:15:10.502912 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.502808 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98974e74-7bfb-4ecd-9f1d-d1cc85d906fc" containerName="main" Sep 11 01:15:10.502912 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.502819 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="98974e74-7bfb-4ecd-9f1d-d1cc85d906fc" containerName="main" Sep 11 01:15:10.502912 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.502837 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63b10eb2-2b57-4a43-abc4-451ffd0e2167" containerName="maas-api" Sep 11 01:15:10.502912 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.502880 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="63b10eb2-2b57-4a43-abc4-451ffd0e2167" containerName="maas-api" Sep 11 01:15:10.503059 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.502945 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="63b10eb2-2b57-4a43-abc4-451ffd0e2167" containerName="maas-api" Sep 11 01:15:10.503059 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.502955 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="98974e74-7bfb-4ecd-9f1d-d1cc85d906fc" containerName="main" Sep 11 01:15:10.507000 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.506979 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:15:10.509727 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.509706 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-651569\"/\"distinct-w4-651569-kserve-self-signed-certs\"" Sep 11 01:15:10.515745 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.515723 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r"] Sep 11 01:15:10.539657 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.539625 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:10.602065 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.602034 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1273e50-f71a-4b79-9ea5-654a64f60134-tmp-dir\") pod \"distinct-w4-651569-kserve-bb5b69ddf-4pg8r\" (UID: \"d1273e50-f71a-4b79-9ea5-654a64f60134\") " pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:15:10.602213 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.602078 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1273e50-f71a-4b79-9ea5-654a64f60134-model-cache\") pod \"distinct-w4-651569-kserve-bb5b69ddf-4pg8r\" (UID: \"d1273e50-f71a-4b79-9ea5-654a64f60134\") " pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:15:10.602213 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.602139 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1273e50-f71a-4b79-9ea5-654a64f60134-home\") pod \"distinct-w4-651569-kserve-bb5b69ddf-4pg8r\" (UID: \"d1273e50-f71a-4b79-9ea5-654a64f60134\") " pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:15:10.602213 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.602165 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1273e50-f71a-4b79-9ea5-654a64f60134-tls-certs\") pod \"distinct-w4-651569-kserve-bb5b69ddf-4pg8r\" (UID: \"d1273e50-f71a-4b79-9ea5-654a64f60134\") " pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:15:10.602213 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.602187 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1273e50-f71a-4b79-9ea5-654a64f60134-dshm\") pod \"distinct-w4-651569-kserve-bb5b69ddf-4pg8r\" (UID: \"d1273e50-f71a-4b79-9ea5-654a64f60134\") " pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:15:10.602399 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.602223 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xb44\" (UniqueName: \"kubernetes.io/projected/d1273e50-f71a-4b79-9ea5-654a64f60134-kube-api-access-5xb44\") pod \"distinct-w4-651569-kserve-bb5b69ddf-4pg8r\" (UID: \"d1273e50-f71a-4b79-9ea5-654a64f60134\") " pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:15:10.702867 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.702816 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xb44\" (UniqueName: \"kubernetes.io/projected/d1273e50-f71a-4b79-9ea5-654a64f60134-kube-api-access-5xb44\") pod \"distinct-w4-651569-kserve-bb5b69ddf-4pg8r\" (UID: \"d1273e50-f71a-4b79-9ea5-654a64f60134\") " pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:15:10.702867 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.702874 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1273e50-f71a-4b79-9ea5-654a64f60134-tmp-dir\") pod \"distinct-w4-651569-kserve-bb5b69ddf-4pg8r\" (UID: \"d1273e50-f71a-4b79-9ea5-654a64f60134\") " pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:15:10.703078 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.702909 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1273e50-f71a-4b79-9ea5-654a64f60134-model-cache\") pod \"distinct-w4-651569-kserve-bb5b69ddf-4pg8r\" (UID: \"d1273e50-f71a-4b79-9ea5-654a64f60134\") " pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:15:10.703078 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.703041 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1273e50-f71a-4b79-9ea5-654a64f60134-home\") pod \"distinct-w4-651569-kserve-bb5b69ddf-4pg8r\" (UID: \"d1273e50-f71a-4b79-9ea5-654a64f60134\") " pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:15:10.703177 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.703090 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1273e50-f71a-4b79-9ea5-654a64f60134-tls-certs\") pod \"distinct-w4-651569-kserve-bb5b69ddf-4pg8r\" (UID: \"d1273e50-f71a-4b79-9ea5-654a64f60134\") " pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:15:10.703177 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.703133 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1273e50-f71a-4b79-9ea5-654a64f60134-dshm\") pod \"distinct-w4-651569-kserve-bb5b69ddf-4pg8r\" (UID: \"d1273e50-f71a-4b79-9ea5-654a64f60134\") " pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:15:10.703353 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.703331 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1273e50-f71a-4b79-9ea5-654a64f60134-tmp-dir\") pod \"distinct-w4-651569-kserve-bb5b69ddf-4pg8r\" (UID: \"d1273e50-f71a-4b79-9ea5-654a64f60134\") " pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:15:10.703353 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.703350 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1273e50-f71a-4b79-9ea5-654a64f60134-model-cache\") pod \"distinct-w4-651569-kserve-bb5b69ddf-4pg8r\" (UID: \"d1273e50-f71a-4b79-9ea5-654a64f60134\") " pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:15:10.703475 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.703410 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1273e50-f71a-4b79-9ea5-654a64f60134-home\") pod \"distinct-w4-651569-kserve-bb5b69ddf-4pg8r\" (UID: \"d1273e50-f71a-4b79-9ea5-654a64f60134\") " pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:15:10.705323 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.705302 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1273e50-f71a-4b79-9ea5-654a64f60134-dshm\") pod \"distinct-w4-651569-kserve-bb5b69ddf-4pg8r\" (UID: \"d1273e50-f71a-4b79-9ea5-654a64f60134\") " pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:15:10.705485 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.705469 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1273e50-f71a-4b79-9ea5-654a64f60134-tls-certs\") pod \"distinct-w4-651569-kserve-bb5b69ddf-4pg8r\" (UID: \"d1273e50-f71a-4b79-9ea5-654a64f60134\") " pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:15:10.711228 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.711206 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xb44\" (UniqueName: \"kubernetes.io/projected/d1273e50-f71a-4b79-9ea5-654a64f60134-kube-api-access-5xb44\") pod \"distinct-w4-651569-kserve-bb5b69ddf-4pg8r\" (UID: \"d1273e50-f71a-4b79-9ea5-654a64f60134\") " pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:15:10.817772 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:10.817701 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:15:11.093004 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:11.092978 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r"] Sep 11 01:15:11.094653 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:15:11.094631 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1273e50_f71a_4b79_9ea5_654a64f60134.slice/crio-d4c9a0481379e7e81bf44c47f13b4fab2e0026050e17b4ace9e7eb96aa042996 WatchSource:0}: Error finding container d4c9a0481379e7e81bf44c47f13b4fab2e0026050e17b4ace9e7eb96aa042996: Status 404 returned error can't find the container with id d4c9a0481379e7e81bf44c47f13b4fab2e0026050e17b4ace9e7eb96aa042996 Sep 11 01:15:11.494459 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:11.494422 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" event={"ID":"d1273e50-f71a-4b79-9ea5-654a64f60134","Type":"ContainerStarted","Data":"41d0b77755cd942d87479922bb5ead3e2af75c5db12c49cd0fb904fe96bad6a9"} Sep 11 01:15:11.494459 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:11.494471 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" event={"ID":"d1273e50-f71a-4b79-9ea5-654a64f60134","Type":"ContainerStarted","Data":"d4c9a0481379e7e81bf44c47f13b4fab2e0026050e17b4ace9e7eb96aa042996"} Sep 11 01:15:11.509888 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:11.509810 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" podStartSLOduration=1.5097917760000001 podStartE2EDuration="1.509791776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:15:11.509737451 +0000 UTC m=+1038.333190961" watchObservedRunningTime="2026-09-11 01:15:11.509791776 +0000 UTC m=+1038.333245287" Sep 11 01:15:14.239228 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.239192 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt"] Sep 11 01:15:14.239638 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.239425 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" podUID="f9896ad1-58bb-4e9e-8aab-276ba405853e" containerName="maas-api" containerID="cri-o://87dab9534814d60e0a375c147c6c7ca79a6e1efdb9eb9e3de1930b98f10fbd55" gracePeriod=30 Sep 11 01:15:14.267832 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.267799 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:14.368196 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.368162 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:14.434805 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.434765 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:14.491180 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.491115 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:14.495822 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.495803 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" Sep 11 01:15:14.512070 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.512035 2554 generic.go:356] "Generic (PLEG): container finished" podID="f9896ad1-58bb-4e9e-8aab-276ba405853e" containerID="87dab9534814d60e0a375c147c6c7ca79a6e1efdb9eb9e3de1930b98f10fbd55" exitCode=0 Sep 11 01:15:14.512290 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.512082 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" Sep 11 01:15:14.512290 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.512123 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" event={"ID":"f9896ad1-58bb-4e9e-8aab-276ba405853e","Type":"ContainerDied","Data":"87dab9534814d60e0a375c147c6c7ca79a6e1efdb9eb9e3de1930b98f10fbd55"} Sep 11 01:15:14.512290 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.512171 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt" event={"ID":"f9896ad1-58bb-4e9e-8aab-276ba405853e","Type":"ContainerDied","Data":"dcaacc5c6a639578ccffdfff8f20707b7cb3cd45ecb95c6a6847c4ae534f40af"} Sep 11 01:15:14.512290 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.512201 2554 scope.go:117] "RemoveContainer" containerID="87dab9534814d60e0a375c147c6c7ca79a6e1efdb9eb9e3de1930b98f10fbd55" Sep 11 01:15:14.523348 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.523325 2554 scope.go:117] "RemoveContainer" containerID="87dab9534814d60e0a375c147c6c7ca79a6e1efdb9eb9e3de1930b98f10fbd55" Sep 11 01:15:14.523638 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:15:14.523618 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87dab9534814d60e0a375c147c6c7ca79a6e1efdb9eb9e3de1930b98f10fbd55\": container with ID starting with 87dab9534814d60e0a375c147c6c7ca79a6e1efdb9eb9e3de1930b98f10fbd55 not found: ID does not exist" containerID="87dab9534814d60e0a375c147c6c7ca79a6e1efdb9eb9e3de1930b98f10fbd55" Sep 11 01:15:14.523700 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.523646 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87dab9534814d60e0a375c147c6c7ca79a6e1efdb9eb9e3de1930b98f10fbd55"} err="failed to get container status \"87dab9534814d60e0a375c147c6c7ca79a6e1efdb9eb9e3de1930b98f10fbd55\": rpc error: code = NotFound desc = could not find container \"87dab9534814d60e0a375c147c6c7ca79a6e1efdb9eb9e3de1930b98f10fbd55\": container with ID starting with 87dab9534814d60e0a375c147c6c7ca79a6e1efdb9eb9e3de1930b98f10fbd55 not found: ID does not exist" Sep 11 01:15:14.534222 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.534197 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9896ad1-58bb-4e9e-8aab-276ba405853e-openshift-service-ca\") pod \"f9896ad1-58bb-4e9e-8aab-276ba405853e\" (UID: \"f9896ad1-58bb-4e9e-8aab-276ba405853e\") " Sep 11 01:15:14.534316 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.534269 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9896ad1-58bb-4e9e-8aab-276ba405853e-maas-api-tls\") pod \"f9896ad1-58bb-4e9e-8aab-276ba405853e\" (UID: \"f9896ad1-58bb-4e9e-8aab-276ba405853e\") " Sep 11 01:15:14.534371 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.534322 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrbmb\" (UniqueName: \"kubernetes.io/projected/f9896ad1-58bb-4e9e-8aab-276ba405853e-kube-api-access-qrbmb\") pod \"f9896ad1-58bb-4e9e-8aab-276ba405853e\" (UID: \"f9896ad1-58bb-4e9e-8aab-276ba405853e\") " Sep 11 01:15:14.534493 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.534470 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9896ad1-58bb-4e9e-8aab-276ba405853e-metrics-certs\") pod \"f9896ad1-58bb-4e9e-8aab-276ba405853e\" (UID: \"f9896ad1-58bb-4e9e-8aab-276ba405853e\") " Sep 11 01:15:14.534579 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.534555 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9896ad1-58bb-4e9e-8aab-276ba405853e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f9896ad1-58bb-4e9e-8aab-276ba405853e" (UID: "f9896ad1-58bb-4e9e-8aab-276ba405853e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:15:14.534865 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.534811 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9896ad1-58bb-4e9e-8aab-276ba405853e-openshift-service-ca\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:15:14.536359 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.536331 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9896ad1-58bb-4e9e-8aab-276ba405853e-kube-api-access-qrbmb" (OuterVolumeSpecName: "kube-api-access-qrbmb") pod "f9896ad1-58bb-4e9e-8aab-276ba405853e" (UID: "f9896ad1-58bb-4e9e-8aab-276ba405853e"). InnerVolumeSpecName "kube-api-access-qrbmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:15:14.536464 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.536449 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9896ad1-58bb-4e9e-8aab-276ba405853e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f9896ad1-58bb-4e9e-8aab-276ba405853e" (UID: "f9896ad1-58bb-4e9e-8aab-276ba405853e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:15:14.536730 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.536705 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9896ad1-58bb-4e9e-8aab-276ba405853e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f9896ad1-58bb-4e9e-8aab-276ba405853e" (UID: "f9896ad1-58bb-4e9e-8aab-276ba405853e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:15:14.635449 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.635422 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9896ad1-58bb-4e9e-8aab-276ba405853e-maas-api-tls\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:15:14.635449 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.635443 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrbmb\" (UniqueName: \"kubernetes.io/projected/f9896ad1-58bb-4e9e-8aab-276ba405853e-kube-api-access-qrbmb\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:15:14.635449 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.635452 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9896ad1-58bb-4e9e-8aab-276ba405853e-metrics-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:15:14.813333 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.813239 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:14.833110 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.833085 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt"] Sep 11 01:15:14.837365 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:14.837334 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b8e72c7-55774554d4-grpdt"] Sep 11 01:15:15.684346 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:15.684314 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9896ad1-58bb-4e9e-8aab-276ba405853e" path="/var/lib/kubelet/pods/f9896ad1-58bb-4e9e-8aab-276ba405853e/volumes" Sep 11 01:15:18.953785 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:18.953722 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:19.313314 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:19.313226 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:20.629796 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:20.629757 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:20.735888 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:20.735809 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:20.817895 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:20.817862 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:15:20.818089 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:20.817915 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:15:20.830649 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:20.830618 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:15:20.909381 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:20.909273 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:21.254458 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:21.254424 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:21.353385 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:21.353352 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:21.475838 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:21.475802 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:21.560093 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:21.560018 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:15:23.299885 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:23.299835 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:24.707941 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:24.707905 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:27.060666 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:27.060627 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:27.226545 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:27.226512 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:27.332093 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:27.331985 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:27.427168 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:27.427137 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:27.580469 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:27.580431 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:33.612986 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:33.612952 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:33.705513 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:33.705471 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:33.802887 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:33.802832 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:34.508126 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:34.508095 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:34.815883 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:34.815783 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:34.982821 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:34.982793 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:37.383137 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:37.383094 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:37.484154 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:37.484122 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:38.693953 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:38.693921 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78"] Sep 11 01:15:38.694528 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:38.694513 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9896ad1-58bb-4e9e-8aab-276ba405853e" containerName="maas-api" Sep 11 01:15:38.694573 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:38.694532 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9896ad1-58bb-4e9e-8aab-276ba405853e" containerName="maas-api" Sep 11 01:15:38.694639 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:38.694628 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9896ad1-58bb-4e9e-8aab-276ba405853e" containerName="maas-api" Sep 11 01:15:38.697956 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:38.697934 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" Sep 11 01:15:38.700485 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:38.700467 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-799002-serving-cert\"" Sep 11 01:15:38.706977 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:38.706954 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78"] Sep 11 01:15:38.708979 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:38.708957 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:38.808364 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:38.808333 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:38.855174 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:38.855145 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f4tt\" (UniqueName: \"kubernetes.io/projected/7d195969-3723-46af-996f-765a6ee5fcc1-kube-api-access-4f4tt\") pod \"maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78\" (UID: \"7d195969-3723-46af-996f-765a6ee5fcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" Sep 11 01:15:38.855449 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:38.855427 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d195969-3723-46af-996f-765a6ee5fcc1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78\" (UID: \"7d195969-3723-46af-996f-765a6ee5fcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" Sep 11 01:15:38.855626 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:38.855610 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d195969-3723-46af-996f-765a6ee5fcc1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78\" (UID: \"7d195969-3723-46af-996f-765a6ee5fcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" Sep 11 01:15:38.855735 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:38.855721 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d195969-3723-46af-996f-765a6ee5fcc1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78\" (UID: \"7d195969-3723-46af-996f-765a6ee5fcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" Sep 11 01:15:38.956396 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:38.956307 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d195969-3723-46af-996f-765a6ee5fcc1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78\" (UID: \"7d195969-3723-46af-996f-765a6ee5fcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" Sep 11 01:15:38.956396 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:38.956353 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d195969-3723-46af-996f-765a6ee5fcc1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78\" (UID: \"7d195969-3723-46af-996f-765a6ee5fcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" Sep 11 01:15:38.956604 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:38.956414 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f4tt\" (UniqueName: \"kubernetes.io/projected/7d195969-3723-46af-996f-765a6ee5fcc1-kube-api-access-4f4tt\") pod \"maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78\" (UID: \"7d195969-3723-46af-996f-765a6ee5fcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" Sep 11 01:15:38.956604 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:15:38.956454 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-serving-cert: secret "maas-api-e2e-shared-a-w2-799002-serving-cert" not found Sep 11 01:15:38.956604 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:38.956491 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d195969-3723-46af-996f-765a6ee5fcc1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78\" (UID: \"7d195969-3723-46af-996f-765a6ee5fcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" Sep 11 01:15:38.956604 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:15:38.956520 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d195969-3723-46af-996f-765a6ee5fcc1-maas-api-tls podName:7d195969-3723-46af-996f-765a6ee5fcc1 nodeName:}" failed. No retries permitted until 2026-09-11 01:15:39.456500364 +0000 UTC m=+1066.279953866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7d195969-3723-46af-996f-765a6ee5fcc1-maas-api-tls") pod "maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" (UID: "7d195969-3723-46af-996f-765a6ee5fcc1") : secret "maas-api-e2e-shared-a-w2-799002-serving-cert" not found Sep 11 01:15:38.957279 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:38.957254 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d195969-3723-46af-996f-765a6ee5fcc1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78\" (UID: \"7d195969-3723-46af-996f-765a6ee5fcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" Sep 11 01:15:38.959078 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:38.959055 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d195969-3723-46af-996f-765a6ee5fcc1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78\" (UID: \"7d195969-3723-46af-996f-765a6ee5fcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" Sep 11 01:15:38.964684 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:38.964660 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f4tt\" (UniqueName: \"kubernetes.io/projected/7d195969-3723-46af-996f-765a6ee5fcc1-kube-api-access-4f4tt\") pod \"maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78\" (UID: \"7d195969-3723-46af-996f-765a6ee5fcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" Sep 11 01:15:39.174912 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:39.174875 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:39.258936 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:39.258857 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:39.460732 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:39.460693 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d195969-3723-46af-996f-765a6ee5fcc1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78\" (UID: \"7d195969-3723-46af-996f-765a6ee5fcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" Sep 11 01:15:39.463217 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:39.463187 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d195969-3723-46af-996f-765a6ee5fcc1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78\" (UID: \"7d195969-3723-46af-996f-765a6ee5fcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" Sep 11 01:15:39.610522 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:39.610453 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" Sep 11 01:15:39.887633 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:39.887605 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78"] Sep 11 01:15:39.889347 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:15:39.889322 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d195969_3723_46af_996f_765a6ee5fcc1.slice/crio-c8975d875a97b12730e512a260e018ff5ba2c7c8a6205f5f024c7da14177b625 WatchSource:0}: Error finding container c8975d875a97b12730e512a260e018ff5ba2c7c8a6205f5f024c7da14177b625: Status 404 returned error can't find the container with id c8975d875a97b12730e512a260e018ff5ba2c7c8a6205f5f024c7da14177b625 Sep 11 01:15:40.656183 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:40.656097 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" event={"ID":"7d195969-3723-46af-996f-765a6ee5fcc1","Type":"ContainerStarted","Data":"7350403fb1fa07ac992ee26daa621586e3b7dc67897c89c707c1c40efba92b52"} Sep 11 01:15:40.656183 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:40.656170 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" event={"ID":"7d195969-3723-46af-996f-765a6ee5fcc1","Type":"ContainerStarted","Data":"c8975d875a97b12730e512a260e018ff5ba2c7c8a6205f5f024c7da14177b625"} Sep 11 01:15:40.656354 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:40.656203 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" Sep 11 01:15:40.672881 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:40.672813 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" podStartSLOduration=2.285433039 podStartE2EDuration="2.672794391s" totalImagesPullingTime="387.361352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:15:40.672709384 +0000 UTC m=+1067.496162894" watchObservedRunningTime="2026-09-11 01:15:40.672794391 +0000 UTC m=+1067.496247902" Sep 11 01:15:46.667203 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:46.667137 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" Sep 11 01:15:51.438926 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:51.438886 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:15:59.604933 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:15:59.604895 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:02.322581 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:02.322545 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:02.651823 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:02.651736 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:02.992192 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:02.992157 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:03.189480 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:03.189448 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:05.615819 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:05.615779 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:05.714407 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:05.714367 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:06.727558 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:06.727524 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278"] Sep 11 01:16:06.732619 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:06.732598 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" Sep 11 01:16:06.735163 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:06.735141 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-71ba6d-serving-cert\"" Sep 11 01:16:06.746201 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:06.746181 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278"] Sep 11 01:16:06.776254 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:06.776222 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:06.804724 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:06.804699 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145ea3e7-21d8-415e-b81c-e534bf889a76-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278\" (UID: \"145ea3e7-21d8-415e-b81c-e534bf889a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" Sep 11 01:16:06.804887 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:06.804740 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145ea3e7-21d8-415e-b81c-e534bf889a76-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278\" (UID: \"145ea3e7-21d8-415e-b81c-e534bf889a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" Sep 11 01:16:06.804887 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:06.804770 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145ea3e7-21d8-415e-b81c-e534bf889a76-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278\" (UID: \"145ea3e7-21d8-415e-b81c-e534bf889a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" Sep 11 01:16:06.805030 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:06.805007 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrqkl\" (UniqueName: \"kubernetes.io/projected/145ea3e7-21d8-415e-b81c-e534bf889a76-kube-api-access-lrqkl\") pod \"maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278\" (UID: \"145ea3e7-21d8-415e-b81c-e534bf889a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" Sep 11 01:16:06.870692 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:06.870669 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:06.905628 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:06.905597 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrqkl\" (UniqueName: \"kubernetes.io/projected/145ea3e7-21d8-415e-b81c-e534bf889a76-kube-api-access-lrqkl\") pod \"maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278\" (UID: \"145ea3e7-21d8-415e-b81c-e534bf889a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" Sep 11 01:16:06.905743 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:06.905659 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145ea3e7-21d8-415e-b81c-e534bf889a76-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278\" (UID: \"145ea3e7-21d8-415e-b81c-e534bf889a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" Sep 11 01:16:06.905743 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:06.905688 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145ea3e7-21d8-415e-b81c-e534bf889a76-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278\" (UID: \"145ea3e7-21d8-415e-b81c-e534bf889a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" Sep 11 01:16:06.905743 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:06.905712 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145ea3e7-21d8-415e-b81c-e534bf889a76-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278\" (UID: \"145ea3e7-21d8-415e-b81c-e534bf889a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" Sep 11 01:16:06.905934 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:16:06.905897 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-serving-cert: secret "maas-api-e2e-shared-b-w2-71ba6d-serving-cert" not found Sep 11 01:16:06.905994 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:16:06.905960 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/145ea3e7-21d8-415e-b81c-e534bf889a76-maas-api-tls podName:145ea3e7-21d8-415e-b81c-e534bf889a76 nodeName:}" failed. No retries permitted until 2026-09-11 01:16:07.405944672 +0000 UTC m=+1094.229398182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/145ea3e7-21d8-415e-b81c-e534bf889a76-maas-api-tls") pod "maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" (UID: "145ea3e7-21d8-415e-b81c-e534bf889a76") : secret "maas-api-e2e-shared-b-w2-71ba6d-serving-cert" not found Sep 11 01:16:06.906385 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:06.906363 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145ea3e7-21d8-415e-b81c-e534bf889a76-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278\" (UID: \"145ea3e7-21d8-415e-b81c-e534bf889a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" Sep 11 01:16:06.908080 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:06.908038 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145ea3e7-21d8-415e-b81c-e534bf889a76-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278\" (UID: \"145ea3e7-21d8-415e-b81c-e534bf889a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" Sep 11 01:16:06.916240 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:06.916214 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrqkl\" (UniqueName: \"kubernetes.io/projected/145ea3e7-21d8-415e-b81c-e534bf889a76-kube-api-access-lrqkl\") pod \"maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278\" (UID: \"145ea3e7-21d8-415e-b81c-e534bf889a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" Sep 11 01:16:07.037182 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:07.037121 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:07.349121 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:07.349034 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:07.410492 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:07.410463 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145ea3e7-21d8-415e-b81c-e534bf889a76-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278\" (UID: \"145ea3e7-21d8-415e-b81c-e534bf889a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" Sep 11 01:16:07.412825 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:07.412803 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145ea3e7-21d8-415e-b81c-e534bf889a76-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278\" (UID: \"145ea3e7-21d8-415e-b81c-e534bf889a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" Sep 11 01:16:07.457793 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:07.457765 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:07.649039 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:07.648964 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" Sep 11 01:16:07.699061 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:07.699019 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:07.934035 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:07.934013 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278"] Sep 11 01:16:07.936362 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:16:07.936337 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145ea3e7_21d8_415e_b81c_e534bf889a76.slice/crio-7cf85b9e16384e73bae89075ed8ac2a3a410fdf988c8f72955f042ad1b792817 WatchSource:0}: Error finding container 7cf85b9e16384e73bae89075ed8ac2a3a410fdf988c8f72955f042ad1b792817: Status 404 returned error can't find the container with id 7cf85b9e16384e73bae89075ed8ac2a3a410fdf988c8f72955f042ad1b792817 Sep 11 01:16:08.218292 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:08.218210 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:08.351450 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:08.351419 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:08.540910 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:08.540811 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:08.803044 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:08.802952 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" event={"ID":"145ea3e7-21d8-415e-b81c-e534bf889a76","Type":"ContainerStarted","Data":"341c1f8a21f94023db4c1e3e8ff62b094120a8037e0bd8ea4bebeb50a54ba020"} Sep 11 01:16:08.803044 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:08.803004 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" event={"ID":"145ea3e7-21d8-415e-b81c-e534bf889a76","Type":"ContainerStarted","Data":"7cf85b9e16384e73bae89075ed8ac2a3a410fdf988c8f72955f042ad1b792817"} Sep 11 01:16:08.803273 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:08.803042 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" Sep 11 01:16:08.822788 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:08.822759 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:08.828271 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:08.828231 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" podStartSLOduration=2.506055686 podStartE2EDuration="2.828217601s" totalImagesPullingTime="322.161915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:16:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:16:08.826230754 +0000 UTC m=+1095.649684255" watchObservedRunningTime="2026-09-11 01:16:08.828217601 +0000 UTC m=+1095.651671111" Sep 11 01:16:09.629077 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:09.629029 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:09.728944 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:09.728910 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:09.867478 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:09.867446 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:14.814122 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:14.814091 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" Sep 11 01:16:28.801235 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:28.801203 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:29.607578 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:29.607535 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:30.601879 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:30.601787 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:36.193491 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.193461 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh"] Sep 11 01:16:36.197156 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.197136 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:16:36.199580 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.199555 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-651569\"/\"unconfigured-w4-651569-kserve-self-signed-certs\"" Sep 11 01:16:36.206390 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.206368 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh"] Sep 11 01:16:36.256183 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.256157 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-dshm\") pod \"unconfigured-w4-651569-kserve-848d7fd45d-9vmzh\" (UID: \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\") " pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:16:36.256288 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.256189 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-tmp-dir\") pod \"unconfigured-w4-651569-kserve-848d7fd45d-9vmzh\" (UID: \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\") " pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:16:36.256288 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.256270 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-home\") pod \"unconfigured-w4-651569-kserve-848d7fd45d-9vmzh\" (UID: \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\") " pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:16:36.256379 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.256315 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-model-cache\") pod \"unconfigured-w4-651569-kserve-848d7fd45d-9vmzh\" (UID: \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\") " pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:16:36.256379 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.256340 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-tls-certs\") pod \"unconfigured-w4-651569-kserve-848d7fd45d-9vmzh\" (UID: \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\") " pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:16:36.256447 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.256426 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql667\" (UniqueName: \"kubernetes.io/projected/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-kube-api-access-ql667\") pod \"unconfigured-w4-651569-kserve-848d7fd45d-9vmzh\" (UID: \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\") " pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:16:36.310663 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.310634 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:36.357507 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.357481 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-home\") pod \"unconfigured-w4-651569-kserve-848d7fd45d-9vmzh\" (UID: \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\") " pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:16:36.357619 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.357527 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-model-cache\") pod \"unconfigured-w4-651569-kserve-848d7fd45d-9vmzh\" (UID: \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\") " pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:16:36.357679 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.357627 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-tls-certs\") pod \"unconfigured-w4-651569-kserve-848d7fd45d-9vmzh\" (UID: \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\") " pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:16:36.357728 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.357689 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql667\" (UniqueName: \"kubernetes.io/projected/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-kube-api-access-ql667\") pod \"unconfigured-w4-651569-kserve-848d7fd45d-9vmzh\" (UID: \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\") " pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:16:36.357802 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.357787 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-dshm\") pod \"unconfigured-w4-651569-kserve-848d7fd45d-9vmzh\" (UID: \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\") " pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:16:36.357888 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.357823 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-tmp-dir\") pod \"unconfigured-w4-651569-kserve-848d7fd45d-9vmzh\" (UID: \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\") " pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:16:36.357953 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.357924 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-home\") pod \"unconfigured-w4-651569-kserve-848d7fd45d-9vmzh\" (UID: \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\") " pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:16:36.358012 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.357962 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-model-cache\") pod \"unconfigured-w4-651569-kserve-848d7fd45d-9vmzh\" (UID: \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\") " pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:16:36.358191 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.358171 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-tmp-dir\") pod \"unconfigured-w4-651569-kserve-848d7fd45d-9vmzh\" (UID: \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\") " pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:16:36.359884 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.359862 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-dshm\") pod \"unconfigured-w4-651569-kserve-848d7fd45d-9vmzh\" (UID: \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\") " pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:16:36.360068 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.360050 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-tls-certs\") pod \"unconfigured-w4-651569-kserve-848d7fd45d-9vmzh\" (UID: \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\") " pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:16:36.365779 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.365761 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql667\" (UniqueName: \"kubernetes.io/projected/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-kube-api-access-ql667\") pod \"unconfigured-w4-651569-kserve-848d7fd45d-9vmzh\" (UID: \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\") " pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:16:36.506886 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.506799 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:16:36.780319 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.780296 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh"] Sep 11 01:16:36.782454 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:16:36.782423 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc47f9ceb_683c_40ff_8a63_8d0d8270b2d1.slice/crio-a4186bad173e5bad8ad35bbe380af899032627c4fddc13c786e1d1adcd3cd956 WatchSource:0}: Error finding container a4186bad173e5bad8ad35bbe380af899032627c4fddc13c786e1d1adcd3cd956: Status 404 returned error can't find the container with id a4186bad173e5bad8ad35bbe380af899032627c4fddc13c786e1d1adcd3cd956 Sep 11 01:16:36.954428 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.954396 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" event={"ID":"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1","Type":"ContainerStarted","Data":"5bc08d3af1555b2cf71ce36c16522f05439643bd44ca9123b83cb8061a086409"} Sep 11 01:16:36.954428 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.954432 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" event={"ID":"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1","Type":"ContainerStarted","Data":"a4186bad173e5bad8ad35bbe380af899032627c4fddc13c786e1d1adcd3cd956"} Sep 11 01:16:36.973627 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:36.973581 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" podStartSLOduration=0.973566872 podStartE2EDuration="973.566872ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:16:36.971067865 +0000 UTC m=+1123.794521386" watchObservedRunningTime="2026-09-11 01:16:36.973566872 +0000 UTC m=+1123.797020412" Sep 11 01:16:38.442293 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:38.442247 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:38.989691 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:38.989658 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg"] Sep 11 01:16:38.989925 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:38.989897 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg" podUID="15915139-3495-41f1-b135-86fd42bf424e" containerName="maas-api" containerID="cri-o://03c601a891760afb6979d2e0c8b45c5ac2406fb072c745d3ce2c494498ae5af3" gracePeriod=30 Sep 11 01:16:39.013474 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.013444 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:39.076616 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.076583 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:39.180452 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.180422 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:39.255685 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.255626 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg" Sep 11 01:16:39.277267 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.277232 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:39.283142 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.283120 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15915139-3495-41f1-b135-86fd42bf424e-openshift-service-ca\") pod \"15915139-3495-41f1-b135-86fd42bf424e\" (UID: \"15915139-3495-41f1-b135-86fd42bf424e\") " Sep 11 01:16:39.283262 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.283193 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15915139-3495-41f1-b135-86fd42bf424e-maas-api-tls\") pod \"15915139-3495-41f1-b135-86fd42bf424e\" (UID: \"15915139-3495-41f1-b135-86fd42bf424e\") " Sep 11 01:16:39.283262 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.283225 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwq2s\" (UniqueName: \"kubernetes.io/projected/15915139-3495-41f1-b135-86fd42bf424e-kube-api-access-rwq2s\") pod \"15915139-3495-41f1-b135-86fd42bf424e\" (UID: \"15915139-3495-41f1-b135-86fd42bf424e\") " Sep 11 01:16:39.283382 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.283304 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15915139-3495-41f1-b135-86fd42bf424e-metrics-certs\") pod \"15915139-3495-41f1-b135-86fd42bf424e\" (UID: \"15915139-3495-41f1-b135-86fd42bf424e\") " Sep 11 01:16:39.283559 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.283532 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15915139-3495-41f1-b135-86fd42bf424e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "15915139-3495-41f1-b135-86fd42bf424e" (UID: "15915139-3495-41f1-b135-86fd42bf424e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:16:39.283730 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.283715 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15915139-3495-41f1-b135-86fd42bf424e-openshift-service-ca\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:16:39.285939 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.285912 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15915139-3495-41f1-b135-86fd42bf424e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "15915139-3495-41f1-b135-86fd42bf424e" (UID: "15915139-3495-41f1-b135-86fd42bf424e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:16:39.285939 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.285930 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15915139-3495-41f1-b135-86fd42bf424e-kube-api-access-rwq2s" (OuterVolumeSpecName: "kube-api-access-rwq2s") pod "15915139-3495-41f1-b135-86fd42bf424e" (UID: "15915139-3495-41f1-b135-86fd42bf424e"). InnerVolumeSpecName "kube-api-access-rwq2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:16:39.286062 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.285987 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15915139-3495-41f1-b135-86fd42bf424e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "15915139-3495-41f1-b135-86fd42bf424e" (UID: "15915139-3495-41f1-b135-86fd42bf424e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:16:39.384913 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.384889 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15915139-3495-41f1-b135-86fd42bf424e-maas-api-tls\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:16:39.384913 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.384910 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwq2s\" (UniqueName: \"kubernetes.io/projected/15915139-3495-41f1-b135-86fd42bf424e-kube-api-access-rwq2s\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:16:39.385050 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.384919 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15915139-3495-41f1-b135-86fd42bf424e-metrics-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:16:39.540061 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.539976 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:39.973538 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.973504 2554 generic.go:356] "Generic (PLEG): container finished" podID="15915139-3495-41f1-b135-86fd42bf424e" containerID="03c601a891760afb6979d2e0c8b45c5ac2406fb072c745d3ce2c494498ae5af3" exitCode=0 Sep 11 01:16:39.973716 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.973550 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg" Sep 11 01:16:39.973716 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.973558 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg" event={"ID":"15915139-3495-41f1-b135-86fd42bf424e","Type":"ContainerDied","Data":"03c601a891760afb6979d2e0c8b45c5ac2406fb072c745d3ce2c494498ae5af3"} Sep 11 01:16:39.974395 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.974159 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg" event={"ID":"15915139-3495-41f1-b135-86fd42bf424e","Type":"ContainerDied","Data":"0c66cfe5bf4203bf8b85d4f9a76fec8cf29c9ebfa827dcc3f9e938fe16114b69"} Sep 11 01:16:39.974395 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.974224 2554 scope.go:117] "RemoveContainer" containerID="03c601a891760afb6979d2e0c8b45c5ac2406fb072c745d3ce2c494498ae5af3" Sep 11 01:16:39.989369 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.989350 2554 scope.go:117] "RemoveContainer" containerID="03c601a891760afb6979d2e0c8b45c5ac2406fb072c745d3ce2c494498ae5af3" Sep 11 01:16:39.989761 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:16:39.989729 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c601a891760afb6979d2e0c8b45c5ac2406fb072c745d3ce2c494498ae5af3\": container with ID starting with 03c601a891760afb6979d2e0c8b45c5ac2406fb072c745d3ce2c494498ae5af3 not found: ID does not exist" containerID="03c601a891760afb6979d2e0c8b45c5ac2406fb072c745d3ce2c494498ae5af3" Sep 11 01:16:39.989837 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.989775 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c601a891760afb6979d2e0c8b45c5ac2406fb072c745d3ce2c494498ae5af3"} err="failed to get container status \"03c601a891760afb6979d2e0c8b45c5ac2406fb072c745d3ce2c494498ae5af3\": rpc error: code = NotFound desc = could not find container \"03c601a891760afb6979d2e0c8b45c5ac2406fb072c745d3ce2c494498ae5af3\": container with ID starting with 03c601a891760afb6979d2e0c8b45c5ac2406fb072c745d3ce2c494498ae5af3 not found: ID does not exist" Sep 11 01:16:39.994118 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.994099 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg"] Sep 11 01:16:39.998529 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:39.998510 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cef92e-5fcd85f6b5-bnvsg"] Sep 11 01:16:40.139414 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:40.139391 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:40.472377 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:40.472340 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:40.754913 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:40.754828 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:41.678358 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:41.678325 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15915139-3495-41f1-b135-86fd42bf424e" path="/var/lib/kubelet/pods/15915139-3495-41f1-b135-86fd42bf424e/volumes" Sep 11 01:16:43.832310 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:43.832284 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:43.928080 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:43.928048 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:44.620963 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:44.620923 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:45.111699 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.111669 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6"] Sep 11 01:16:45.112879 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.112819 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15915139-3495-41f1-b135-86fd42bf424e" containerName="maas-api" Sep 11 01:16:45.113057 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.113044 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="15915139-3495-41f1-b135-86fd42bf424e" containerName="maas-api" Sep 11 01:16:45.113280 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.113265 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="15915139-3495-41f1-b135-86fd42bf424e" containerName="maas-api" Sep 11 01:16:45.118394 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.118363 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6"] Sep 11 01:16:45.118589 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.118509 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6" Sep 11 01:16:45.121163 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.121141 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-194eb64c-serving-cert\"" Sep 11 01:16:45.132992 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.132967 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:45.233920 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.233888 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:45.237009 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.236988 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxjkt\" (UniqueName: \"kubernetes.io/projected/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb-kube-api-access-nxjkt\") pod \"maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6\" (UID: \"39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6" Sep 11 01:16:45.237112 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.237015 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb-openshift-service-ca\") pod \"maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6\" (UID: \"39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6" Sep 11 01:16:45.237112 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.237091 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb-metrics-certs\") pod \"maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6\" (UID: \"39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6" Sep 11 01:16:45.237222 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.237199 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb-maas-api-tls\") pod \"maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6\" (UID: \"39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6" Sep 11 01:16:45.334536 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.334507 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:45.337702 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.337681 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb-metrics-certs\") pod \"maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6\" (UID: \"39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6" Sep 11 01:16:45.337789 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.337744 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb-maas-api-tls\") pod \"maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6\" (UID: \"39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6" Sep 11 01:16:45.337789 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.337765 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxjkt\" (UniqueName: \"kubernetes.io/projected/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb-kube-api-access-nxjkt\") pod \"maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6\" (UID: \"39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6" Sep 11 01:16:45.337789 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.337782 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb-openshift-service-ca\") pod \"maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6\" (UID: \"39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6" Sep 11 01:16:45.338475 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.338455 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb-openshift-service-ca\") pod \"maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6\" (UID: \"39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6" Sep 11 01:16:45.340299 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.340275 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb-metrics-certs\") pod \"maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6\" (UID: \"39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6" Sep 11 01:16:45.340405 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.340275 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb-maas-api-tls\") pod \"maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6\" (UID: \"39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6" Sep 11 01:16:45.345677 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.345655 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxjkt\" (UniqueName: \"kubernetes.io/projected/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb-kube-api-access-nxjkt\") pod \"maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6\" (UID: \"39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6" Sep 11 01:16:45.430364 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.430345 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6" Sep 11 01:16:45.680078 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.680047 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:45.706490 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.706465 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6"] Sep 11 01:16:45.708706 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:16:45.708675 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d50f6e_ebe6_4b8a_9ff7_46a1f2c9ebcb.slice/crio-db2c50c1a6193175c8464d86947ebd8536de33e10dcc1d28dddd68f6920ff35c WatchSource:0}: Error finding container db2c50c1a6193175c8464d86947ebd8536de33e10dcc1d28dddd68f6920ff35c: Status 404 returned error can't find the container with id db2c50c1a6193175c8464d86947ebd8536de33e10dcc1d28dddd68f6920ff35c Sep 11 01:16:45.772545 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:45.772485 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:46.006029 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:46.005952 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6" event={"ID":"39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb","Type":"ContainerStarted","Data":"db2c50c1a6193175c8464d86947ebd8536de33e10dcc1d28dddd68f6920ff35c"} Sep 11 01:16:46.507410 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:46.507374 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:16:46.507754 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:46.507426 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:16:46.519935 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:46.519914 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:16:47.012911 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:47.012873 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6" event={"ID":"39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb","Type":"ContainerStarted","Data":"58b6ed35be2753994476f3967ee8917c4bc05008b3a8cd3c727a971ed59a888a"} Sep 11 01:16:47.013077 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:47.012989 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6" Sep 11 01:16:47.024821 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:47.024801 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:16:47.032041 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:47.032001 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6" podStartSLOduration=1.68171963 podStartE2EDuration="2.031986988s" totalImagesPullingTime="350.267358ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:16:47.029338509 +0000 UTC m=+1133.852792019" watchObservedRunningTime="2026-09-11 01:16:47.031986988 +0000 UTC m=+1133.855440498" Sep 11 01:16:52.761956 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:52.761921 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:16:53.023807 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:16:53.023739 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6" Sep 11 01:17:06.996127 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:06.996088 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:17:08.845025 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:08.844986 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:17:18.843044 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:18.842955 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:17:18.959179 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:18.959151 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:17:21.131490 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:21.131451 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:17:21.506542 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:21.506510 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:17:21.988762 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:21.988725 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:17:31.968396 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:31.968364 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:17:32.699809 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:32.699773 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:17:40.282404 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.282363 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw"] Sep 11 01:17:40.286077 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.286060 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:17:40.288401 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.288381 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6ba087\"/\"distinct-w0-6ba087-kserve-self-signed-certs\"" Sep 11 01:17:40.294141 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.294119 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw"] Sep 11 01:17:40.322297 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.322275 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2093b37-1516-4fdf-a834-f6c039b0b05c-tls-certs\") pod \"distinct-w0-6ba087-kserve-5b67658479-69vpw\" (UID: \"f2093b37-1516-4fdf-a834-f6c039b0b05c\") " pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:17:40.322393 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.322315 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2093b37-1516-4fdf-a834-f6c039b0b05c-tmp-dir\") pod \"distinct-w0-6ba087-kserve-5b67658479-69vpw\" (UID: \"f2093b37-1516-4fdf-a834-f6c039b0b05c\") " pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:17:40.322393 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.322334 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2093b37-1516-4fdf-a834-f6c039b0b05c-dshm\") pod \"distinct-w0-6ba087-kserve-5b67658479-69vpw\" (UID: \"f2093b37-1516-4fdf-a834-f6c039b0b05c\") " pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:17:40.322393 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.322379 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2093b37-1516-4fdf-a834-f6c039b0b05c-model-cache\") pod \"distinct-w0-6ba087-kserve-5b67658479-69vpw\" (UID: \"f2093b37-1516-4fdf-a834-f6c039b0b05c\") " pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:17:40.322502 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.322397 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdl26\" (UniqueName: \"kubernetes.io/projected/f2093b37-1516-4fdf-a834-f6c039b0b05c-kube-api-access-zdl26\") pod \"distinct-w0-6ba087-kserve-5b67658479-69vpw\" (UID: \"f2093b37-1516-4fdf-a834-f6c039b0b05c\") " pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:17:40.322502 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.322450 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2093b37-1516-4fdf-a834-f6c039b0b05c-home\") pod \"distinct-w0-6ba087-kserve-5b67658479-69vpw\" (UID: \"f2093b37-1516-4fdf-a834-f6c039b0b05c\") " pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:17:40.407009 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.406979 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:17:40.423123 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.423097 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2093b37-1516-4fdf-a834-f6c039b0b05c-dshm\") pod \"distinct-w0-6ba087-kserve-5b67658479-69vpw\" (UID: \"f2093b37-1516-4fdf-a834-f6c039b0b05c\") " pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:17:40.423228 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.423132 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2093b37-1516-4fdf-a834-f6c039b0b05c-model-cache\") pod \"distinct-w0-6ba087-kserve-5b67658479-69vpw\" (UID: \"f2093b37-1516-4fdf-a834-f6c039b0b05c\") " pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:17:40.423228 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.423155 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdl26\" (UniqueName: \"kubernetes.io/projected/f2093b37-1516-4fdf-a834-f6c039b0b05c-kube-api-access-zdl26\") pod \"distinct-w0-6ba087-kserve-5b67658479-69vpw\" (UID: \"f2093b37-1516-4fdf-a834-f6c039b0b05c\") " pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:17:40.423303 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.423229 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2093b37-1516-4fdf-a834-f6c039b0b05c-home\") pod \"distinct-w0-6ba087-kserve-5b67658479-69vpw\" (UID: \"f2093b37-1516-4fdf-a834-f6c039b0b05c\") " pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:17:40.423346 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.423303 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2093b37-1516-4fdf-a834-f6c039b0b05c-tls-certs\") pod \"distinct-w0-6ba087-kserve-5b67658479-69vpw\" (UID: \"f2093b37-1516-4fdf-a834-f6c039b0b05c\") " pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:17:40.423442 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.423360 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2093b37-1516-4fdf-a834-f6c039b0b05c-tmp-dir\") pod \"distinct-w0-6ba087-kserve-5b67658479-69vpw\" (UID: \"f2093b37-1516-4fdf-a834-f6c039b0b05c\") " pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:17:40.423623 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.423604 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2093b37-1516-4fdf-a834-f6c039b0b05c-model-cache\") pod \"distinct-w0-6ba087-kserve-5b67658479-69vpw\" (UID: \"f2093b37-1516-4fdf-a834-f6c039b0b05c\") " pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:17:40.423695 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.423638 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2093b37-1516-4fdf-a834-f6c039b0b05c-home\") pod \"distinct-w0-6ba087-kserve-5b67658479-69vpw\" (UID: \"f2093b37-1516-4fdf-a834-f6c039b0b05c\") " pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:17:40.423695 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.423675 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2093b37-1516-4fdf-a834-f6c039b0b05c-tmp-dir\") pod \"distinct-w0-6ba087-kserve-5b67658479-69vpw\" (UID: \"f2093b37-1516-4fdf-a834-f6c039b0b05c\") " pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:17:40.425392 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.425373 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2093b37-1516-4fdf-a834-f6c039b0b05c-dshm\") pod \"distinct-w0-6ba087-kserve-5b67658479-69vpw\" (UID: \"f2093b37-1516-4fdf-a834-f6c039b0b05c\") " pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:17:40.425656 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.425638 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2093b37-1516-4fdf-a834-f6c039b0b05c-tls-certs\") pod \"distinct-w0-6ba087-kserve-5b67658479-69vpw\" (UID: \"f2093b37-1516-4fdf-a834-f6c039b0b05c\") " pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:17:40.431816 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.431797 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdl26\" (UniqueName: \"kubernetes.io/projected/f2093b37-1516-4fdf-a834-f6c039b0b05c-kube-api-access-zdl26\") pod \"distinct-w0-6ba087-kserve-5b67658479-69vpw\" (UID: \"f2093b37-1516-4fdf-a834-f6c039b0b05c\") " pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:17:40.596409 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.596343 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:17:40.872259 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:40.872233 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw"] Sep 11 01:17:40.873552 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:17:40.873529 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2093b37_1516_4fdf_a834_f6c039b0b05c.slice/crio-983e4f3c26237658ef84a19d482bd90871e76b9806956671b137bd8579ffe753 WatchSource:0}: Error finding container 983e4f3c26237658ef84a19d482bd90871e76b9806956671b137bd8579ffe753: Status 404 returned error can't find the container with id 983e4f3c26237658ef84a19d482bd90871e76b9806956671b137bd8579ffe753 Sep 11 01:17:41.313470 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:41.313431 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" event={"ID":"f2093b37-1516-4fdf-a834-f6c039b0b05c","Type":"ContainerStarted","Data":"f4b33256a3d7e92f1065a0fdf1e257cf5b7978567d501ba6fe569b441ae7ee29"} Sep 11 01:17:41.313470 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:41.313473 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" event={"ID":"f2093b37-1516-4fdf-a834-f6c039b0b05c","Type":"ContainerStarted","Data":"983e4f3c26237658ef84a19d482bd90871e76b9806956671b137bd8579ffe753"} Sep 11 01:17:41.332456 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:41.332407 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" podStartSLOduration=1.332392434 podStartE2EDuration="1.332392434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:17:41.329704656 +0000 UTC m=+1188.153158167" watchObservedRunningTime="2026-09-11 01:17:41.332392434 +0000 UTC m=+1188.155845944" Sep 11 01:17:50.597192 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:50.597143 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:17:50.597192 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:50.597204 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:17:50.613609 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:50.613583 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:17:51.380186 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:51.380158 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:17:53.669142 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:53.669105 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wtd8v_c2b320dd-43e9-435d-8e4f-2bca1f1233ff/ovn-acl-logging/0.log" Sep 11 01:17:53.670593 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:53.670569 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wtd8v_c2b320dd-43e9-435d-8e4f-2bca1f1233ff/ovn-acl-logging/0.log" Sep 11 01:17:58.547006 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:58.546965 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:17:58.938376 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:17:58.938342 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:18:03.520580 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:03.520545 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw"] Sep 11 01:18:03.536969 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:03.536940 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2"] Sep 11 01:18:03.614664 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:03.614634 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:18:09.792600 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:09.792567 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:18:10.241094 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:10.241055 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:18:14.212127 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.212094 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d"] Sep 11 01:18:14.212636 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.212323 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" podUID="eaa119bd-3d65-4667-b2f3-50e7ac3dd00f" containerName="maas-api" containerID="cri-o://15454a02b38559e09a1e95352d6775a3a78b049aa471b98269dd1469439001ae" gracePeriod=30 Sep 11 01:18:14.241863 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.241795 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:18:14.374648 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.374598 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:18:14.466559 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.466508 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" Sep 11 01:18:14.487898 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.487864 2554 generic.go:356] "Generic (PLEG): container finished" podID="eaa119bd-3d65-4667-b2f3-50e7ac3dd00f" containerID="15454a02b38559e09a1e95352d6775a3a78b049aa471b98269dd1469439001ae" exitCode=0 Sep 11 01:18:14.487898 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.487895 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" Sep 11 01:18:14.488046 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.487940 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" event={"ID":"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f","Type":"ContainerDied","Data":"15454a02b38559e09a1e95352d6775a3a78b049aa471b98269dd1469439001ae"} Sep 11 01:18:14.488046 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.488004 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d" event={"ID":"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f","Type":"ContainerDied","Data":"9225834b37acaf3b96a1a692074595d79910dd2dfa64468b96a5ffdda2784261"} Sep 11 01:18:14.488046 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.488034 2554 scope.go:117] "RemoveContainer" containerID="15454a02b38559e09a1e95352d6775a3a78b049aa471b98269dd1469439001ae" Sep 11 01:18:14.497063 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.497045 2554 scope.go:117] "RemoveContainer" containerID="15454a02b38559e09a1e95352d6775a3a78b049aa471b98269dd1469439001ae" Sep 11 01:18:14.497316 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:18:14.497295 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15454a02b38559e09a1e95352d6775a3a78b049aa471b98269dd1469439001ae\": container with ID starting with 15454a02b38559e09a1e95352d6775a3a78b049aa471b98269dd1469439001ae not found: ID does not exist" containerID="15454a02b38559e09a1e95352d6775a3a78b049aa471b98269dd1469439001ae" Sep 11 01:18:14.497372 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.497324 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15454a02b38559e09a1e95352d6775a3a78b049aa471b98269dd1469439001ae"} err="failed to get container status \"15454a02b38559e09a1e95352d6775a3a78b049aa471b98269dd1469439001ae\": rpc error: code = NotFound desc = could not find container \"15454a02b38559e09a1e95352d6775a3a78b049aa471b98269dd1469439001ae\": container with ID starting with 15454a02b38559e09a1e95352d6775a3a78b049aa471b98269dd1469439001ae not found: ID does not exist" Sep 11 01:18:14.518232 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.518208 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:18:14.593857 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.593815 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:18:14.626728 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.626700 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-openshift-service-ca\") pod \"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f\" (UID: \"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f\") " Sep 11 01:18:14.626930 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.626735 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-maas-api-tls\") pod \"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f\" (UID: \"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f\") " Sep 11 01:18:14.626930 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.626813 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfblv\" (UniqueName: \"kubernetes.io/projected/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-kube-api-access-rfblv\") pod \"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f\" (UID: \"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f\") " Sep 11 01:18:14.626930 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.626912 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-metrics-certs\") pod \"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f\" (UID: \"eaa119bd-3d65-4667-b2f3-50e7ac3dd00f\") " Sep 11 01:18:14.627109 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.627030 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eaa119bd-3d65-4667-b2f3-50e7ac3dd00f" (UID: "eaa119bd-3d65-4667-b2f3-50e7ac3dd00f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:18:14.627269 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.627253 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-openshift-service-ca\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:14.629157 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.629129 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "eaa119bd-3d65-4667-b2f3-50e7ac3dd00f" (UID: "eaa119bd-3d65-4667-b2f3-50e7ac3dd00f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:18:14.629157 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.629144 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-kube-api-access-rfblv" (OuterVolumeSpecName: "kube-api-access-rfblv") pod "eaa119bd-3d65-4667-b2f3-50e7ac3dd00f" (UID: "eaa119bd-3d65-4667-b2f3-50e7ac3dd00f"). InnerVolumeSpecName "kube-api-access-rfblv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:18:14.629323 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.629244 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eaa119bd-3d65-4667-b2f3-50e7ac3dd00f" (UID: "eaa119bd-3d65-4667-b2f3-50e7ac3dd00f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:18:14.728308 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.728256 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfblv\" (UniqueName: \"kubernetes.io/projected/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-kube-api-access-rfblv\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:14.728308 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.728278 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-metrics-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:14.728308 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.728286 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f-maas-api-tls\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:14.813076 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.813052 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d"] Sep 11 01:18:14.815135 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.815114 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ba087-6b89dc5978-j2g8d"] Sep 11 01:18:14.953763 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:14.953733 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:18:15.678483 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:15.678448 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaa119bd-3d65-4667-b2f3-50e7ac3dd00f" path="/var/lib/kubelet/pods/eaa119bd-3d65-4667-b2f3-50e7ac3dd00f/volumes" Sep 11 01:18:18.540241 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.540167 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" podUID="f2093b37-1516-4fdf-a834-f6c039b0b05c" containerName="main" containerID="cri-o://f4b33256a3d7e92f1065a0fdf1e257cf5b7978567d501ba6fe569b441ae7ee29" gracePeriod=45 Sep 11 01:18:18.556032 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.556003 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" podUID="2738da06-3225-42c9-b77a-86e864a9babd" containerName="main" containerID="cri-o://6575f777023a500f0dcae8fa7dff8a2f859df666f5e1a246d44483ab57e81479" gracePeriod=45 Sep 11 01:18:18.818322 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.818278 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:18:18.835003 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.834764 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:18:18.964935 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.964908 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2093b37-1516-4fdf-a834-f6c039b0b05c-tmp-dir\") pod \"f2093b37-1516-4fdf-a834-f6c039b0b05c\" (UID: \"f2093b37-1516-4fdf-a834-f6c039b0b05c\") " Sep 11 01:18:18.965093 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.964940 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2738da06-3225-42c9-b77a-86e864a9babd-model-cache\") pod \"2738da06-3225-42c9-b77a-86e864a9babd\" (UID: \"2738da06-3225-42c9-b77a-86e864a9babd\") " Sep 11 01:18:18.965093 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.964963 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2093b37-1516-4fdf-a834-f6c039b0b05c-tls-certs\") pod \"f2093b37-1516-4fdf-a834-f6c039b0b05c\" (UID: \"f2093b37-1516-4fdf-a834-f6c039b0b05c\") " Sep 11 01:18:18.965093 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.964981 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2093b37-1516-4fdf-a834-f6c039b0b05c-home\") pod \"f2093b37-1516-4fdf-a834-f6c039b0b05c\" (UID: \"f2093b37-1516-4fdf-a834-f6c039b0b05c\") " Sep 11 01:18:18.965093 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.965019 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2738da06-3225-42c9-b77a-86e864a9babd-home\") pod \"2738da06-3225-42c9-b77a-86e864a9babd\" (UID: \"2738da06-3225-42c9-b77a-86e864a9babd\") " Sep 11 01:18:18.965093 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.965074 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmvt4\" (UniqueName: \"kubernetes.io/projected/2738da06-3225-42c9-b77a-86e864a9babd-kube-api-access-gmvt4\") pod \"2738da06-3225-42c9-b77a-86e864a9babd\" (UID: \"2738da06-3225-42c9-b77a-86e864a9babd\") " Sep 11 01:18:18.965345 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.965109 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2093b37-1516-4fdf-a834-f6c039b0b05c-model-cache\") pod \"f2093b37-1516-4fdf-a834-f6c039b0b05c\" (UID: \"f2093b37-1516-4fdf-a834-f6c039b0b05c\") " Sep 11 01:18:18.965345 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.965137 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2738da06-3225-42c9-b77a-86e864a9babd-tls-certs\") pod \"2738da06-3225-42c9-b77a-86e864a9babd\" (UID: \"2738da06-3225-42c9-b77a-86e864a9babd\") " Sep 11 01:18:18.965345 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.965162 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2738da06-3225-42c9-b77a-86e864a9babd-dshm\") pod \"2738da06-3225-42c9-b77a-86e864a9babd\" (UID: \"2738da06-3225-42c9-b77a-86e864a9babd\") " Sep 11 01:18:18.965345 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.965189 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2738da06-3225-42c9-b77a-86e864a9babd-tmp-dir\") pod \"2738da06-3225-42c9-b77a-86e864a9babd\" (UID: \"2738da06-3225-42c9-b77a-86e864a9babd\") " Sep 11 01:18:18.965345 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.965218 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2093b37-1516-4fdf-a834-f6c039b0b05c-home" (OuterVolumeSpecName: "home") pod "f2093b37-1516-4fdf-a834-f6c039b0b05c" (UID: "f2093b37-1516-4fdf-a834-f6c039b0b05c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:18:18.965345 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.965223 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2738da06-3225-42c9-b77a-86e864a9babd-model-cache" (OuterVolumeSpecName: "model-cache") pod "2738da06-3225-42c9-b77a-86e864a9babd" (UID: "2738da06-3225-42c9-b77a-86e864a9babd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:18:18.965345 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.965268 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2738da06-3225-42c9-b77a-86e864a9babd-home" (OuterVolumeSpecName: "home") pod "2738da06-3225-42c9-b77a-86e864a9babd" (UID: "2738da06-3225-42c9-b77a-86e864a9babd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:18:18.965345 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.965227 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2093b37-1516-4fdf-a834-f6c039b0b05c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2093b37-1516-4fdf-a834-f6c039b0b05c" (UID: "f2093b37-1516-4fdf-a834-f6c039b0b05c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:18:18.965345 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.965225 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2093b37-1516-4fdf-a834-f6c039b0b05c-dshm\") pod \"f2093b37-1516-4fdf-a834-f6c039b0b05c\" (UID: \"f2093b37-1516-4fdf-a834-f6c039b0b05c\") " Sep 11 01:18:18.965345 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.965340 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdl26\" (UniqueName: \"kubernetes.io/projected/f2093b37-1516-4fdf-a834-f6c039b0b05c-kube-api-access-zdl26\") pod \"f2093b37-1516-4fdf-a834-f6c039b0b05c\" (UID: \"f2093b37-1516-4fdf-a834-f6c039b0b05c\") " Sep 11 01:18:18.965975 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.965440 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2738da06-3225-42c9-b77a-86e864a9babd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2738da06-3225-42c9-b77a-86e864a9babd" (UID: "2738da06-3225-42c9-b77a-86e864a9babd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:18:18.965975 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.965449 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2093b37-1516-4fdf-a834-f6c039b0b05c-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2093b37-1516-4fdf-a834-f6c039b0b05c" (UID: "f2093b37-1516-4fdf-a834-f6c039b0b05c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:18:18.965975 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.965728 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2093b37-1516-4fdf-a834-f6c039b0b05c-tmp-dir\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:18.965975 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.965754 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2738da06-3225-42c9-b77a-86e864a9babd-model-cache\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:18.965975 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.965772 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2093b37-1516-4fdf-a834-f6c039b0b05c-home\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:18.965975 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.965786 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2738da06-3225-42c9-b77a-86e864a9babd-home\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:18.965975 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.965798 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2093b37-1516-4fdf-a834-f6c039b0b05c-model-cache\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:18.965975 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.965810 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2738da06-3225-42c9-b77a-86e864a9babd-tmp-dir\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:18.967607 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.967583 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2738da06-3225-42c9-b77a-86e864a9babd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2738da06-3225-42c9-b77a-86e864a9babd" (UID: "2738da06-3225-42c9-b77a-86e864a9babd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:18:18.967607 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.967590 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2093b37-1516-4fdf-a834-f6c039b0b05c-kube-api-access-zdl26" (OuterVolumeSpecName: "kube-api-access-zdl26") pod "f2093b37-1516-4fdf-a834-f6c039b0b05c" (UID: "f2093b37-1516-4fdf-a834-f6c039b0b05c"). InnerVolumeSpecName "kube-api-access-zdl26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:18:18.967751 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.967708 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2093b37-1516-4fdf-a834-f6c039b0b05c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2093b37-1516-4fdf-a834-f6c039b0b05c" (UID: "f2093b37-1516-4fdf-a834-f6c039b0b05c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:18:18.967751 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.967706 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2738da06-3225-42c9-b77a-86e864a9babd-kube-api-access-gmvt4" (OuterVolumeSpecName: "kube-api-access-gmvt4") pod "2738da06-3225-42c9-b77a-86e864a9babd" (UID: "2738da06-3225-42c9-b77a-86e864a9babd"). InnerVolumeSpecName "kube-api-access-gmvt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:18:18.967751 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.967723 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2738da06-3225-42c9-b77a-86e864a9babd-dshm" (OuterVolumeSpecName: "dshm") pod "2738da06-3225-42c9-b77a-86e864a9babd" (UID: "2738da06-3225-42c9-b77a-86e864a9babd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:18:18.967751 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:18.967733 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2093b37-1516-4fdf-a834-f6c039b0b05c-dshm" (OuterVolumeSpecName: "dshm") pod "f2093b37-1516-4fdf-a834-f6c039b0b05c" (UID: "f2093b37-1516-4fdf-a834-f6c039b0b05c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:18:19.046858 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.046773 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:18:19.066332 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.066307 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2093b37-1516-4fdf-a834-f6c039b0b05c-tls-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:19.066332 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.066327 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmvt4\" (UniqueName: \"kubernetes.io/projected/2738da06-3225-42c9-b77a-86e864a9babd-kube-api-access-gmvt4\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:19.066509 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.066337 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2738da06-3225-42c9-b77a-86e864a9babd-tls-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:19.066509 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.066346 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2738da06-3225-42c9-b77a-86e864a9babd-dshm\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:19.066509 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.066353 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2093b37-1516-4fdf-a834-f6c039b0b05c-dshm\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:19.066509 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.066360 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdl26\" (UniqueName: \"kubernetes.io/projected/f2093b37-1516-4fdf-a834-f6c039b0b05c-kube-api-access-zdl26\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:19.517511 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.517473 2554 generic.go:356] "Generic (PLEG): container finished" podID="f2093b37-1516-4fdf-a834-f6c039b0b05c" containerID="f4b33256a3d7e92f1065a0fdf1e257cf5b7978567d501ba6fe569b441ae7ee29" exitCode=0 Sep 11 01:18:19.517667 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.517517 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" Sep 11 01:18:19.517667 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.517553 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" event={"ID":"f2093b37-1516-4fdf-a834-f6c039b0b05c","Type":"ContainerDied","Data":"f4b33256a3d7e92f1065a0fdf1e257cf5b7978567d501ba6fe569b441ae7ee29"} Sep 11 01:18:19.517667 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.517598 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw" event={"ID":"f2093b37-1516-4fdf-a834-f6c039b0b05c","Type":"ContainerDied","Data":"983e4f3c26237658ef84a19d482bd90871e76b9806956671b137bd8579ffe753"} Sep 11 01:18:19.517667 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.517615 2554 scope.go:117] "RemoveContainer" containerID="f4b33256a3d7e92f1065a0fdf1e257cf5b7978567d501ba6fe569b441ae7ee29" Sep 11 01:18:19.519216 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.519191 2554 generic.go:356] "Generic (PLEG): container finished" podID="2738da06-3225-42c9-b77a-86e864a9babd" containerID="6575f777023a500f0dcae8fa7dff8a2f859df666f5e1a246d44483ab57e81479" exitCode=0 Sep 11 01:18:19.519328 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.519226 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" Sep 11 01:18:19.519328 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.519253 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" event={"ID":"2738da06-3225-42c9-b77a-86e864a9babd","Type":"ContainerDied","Data":"6575f777023a500f0dcae8fa7dff8a2f859df666f5e1a246d44483ab57e81479"} Sep 11 01:18:19.519328 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.519285 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2" event={"ID":"2738da06-3225-42c9-b77a-86e864a9babd","Type":"ContainerDied","Data":"0f7cf5bbb8181d7ed1cadfab4ca1774958d894617e7f212008a482f149d5e53d"} Sep 11 01:18:19.527753 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.527732 2554 scope.go:117] "RemoveContainer" containerID="f4b33256a3d7e92f1065a0fdf1e257cf5b7978567d501ba6fe569b441ae7ee29" Sep 11 01:18:19.528123 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:18:19.528103 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b33256a3d7e92f1065a0fdf1e257cf5b7978567d501ba6fe569b441ae7ee29\": container with ID starting with f4b33256a3d7e92f1065a0fdf1e257cf5b7978567d501ba6fe569b441ae7ee29 not found: ID does not exist" containerID="f4b33256a3d7e92f1065a0fdf1e257cf5b7978567d501ba6fe569b441ae7ee29" Sep 11 01:18:19.528218 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.528129 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b33256a3d7e92f1065a0fdf1e257cf5b7978567d501ba6fe569b441ae7ee29"} err="failed to get container status \"f4b33256a3d7e92f1065a0fdf1e257cf5b7978567d501ba6fe569b441ae7ee29\": rpc error: code = NotFound desc = could not find container \"f4b33256a3d7e92f1065a0fdf1e257cf5b7978567d501ba6fe569b441ae7ee29\": container with ID starting with f4b33256a3d7e92f1065a0fdf1e257cf5b7978567d501ba6fe569b441ae7ee29 not found: ID does not exist" Sep 11 01:18:19.528218 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.528146 2554 scope.go:117] "RemoveContainer" containerID="6575f777023a500f0dcae8fa7dff8a2f859df666f5e1a246d44483ab57e81479" Sep 11 01:18:19.536013 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.535995 2554 scope.go:117] "RemoveContainer" containerID="6575f777023a500f0dcae8fa7dff8a2f859df666f5e1a246d44483ab57e81479" Sep 11 01:18:19.536266 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:18:19.536249 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6575f777023a500f0dcae8fa7dff8a2f859df666f5e1a246d44483ab57e81479\": container with ID starting with 6575f777023a500f0dcae8fa7dff8a2f859df666f5e1a246d44483ab57e81479 not found: ID does not exist" containerID="6575f777023a500f0dcae8fa7dff8a2f859df666f5e1a246d44483ab57e81479" Sep 11 01:18:19.536322 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.536271 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6575f777023a500f0dcae8fa7dff8a2f859df666f5e1a246d44483ab57e81479"} err="failed to get container status \"6575f777023a500f0dcae8fa7dff8a2f859df666f5e1a246d44483ab57e81479\": rpc error: code = NotFound desc = could not find container \"6575f777023a500f0dcae8fa7dff8a2f859df666f5e1a246d44483ab57e81479\": container with ID starting with 6575f777023a500f0dcae8fa7dff8a2f859df666f5e1a246d44483ab57e81479 not found: ID does not exist" Sep 11 01:18:19.540383 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.540364 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw"] Sep 11 01:18:19.545017 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.544998 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6ba087/distinct-w0-6ba087-kserve-5b67658479-69vpw"] Sep 11 01:18:19.556261 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.556159 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2"] Sep 11 01:18:19.560126 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.560101 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6ba087/simulator-w0-6ba087-kserve-5bbd44496f-cmpn2"] Sep 11 01:18:19.678754 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.678727 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2738da06-3225-42c9-b77a-86e864a9babd" path="/var/lib/kubelet/pods/2738da06-3225-42c9-b77a-86e864a9babd/volumes" Sep 11 01:18:19.679154 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:19.679140 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2093b37-1516-4fdf-a834-f6c039b0b05c" path="/var/lib/kubelet/pods/f2093b37-1516-4fdf-a834-f6c039b0b05c/volumes" Sep 11 01:18:20.628767 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:20.628732 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:18:20.690252 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:20.690220 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2"] Sep 11 01:18:20.690586 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:20.690553 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2" podUID="9356eb13-4c12-4d62-b8f7-d0ec958fcc4b" containerName="maas-api" containerID="cri-o://22ac1554a4f60f14b0dab08ff7676688baf2416f040acdd4a95c3bb99295cb49" gracePeriod=30 Sep 11 01:18:20.720521 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:20.720494 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:18:20.799141 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:20.799105 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:18:20.878310 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:20.878224 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:18:20.944112 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:20.944075 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:18:20.960187 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:20.960169 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2" Sep 11 01:18:20.983269 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:20.983248 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b-openshift-service-ca\") pod \"9356eb13-4c12-4d62-b8f7-d0ec958fcc4b\" (UID: \"9356eb13-4c12-4d62-b8f7-d0ec958fcc4b\") " Sep 11 01:18:20.983390 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:20.983303 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b-metrics-certs\") pod \"9356eb13-4c12-4d62-b8f7-d0ec958fcc4b\" (UID: \"9356eb13-4c12-4d62-b8f7-d0ec958fcc4b\") " Sep 11 01:18:20.983390 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:20.983338 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b-maas-api-tls\") pod \"9356eb13-4c12-4d62-b8f7-d0ec958fcc4b\" (UID: \"9356eb13-4c12-4d62-b8f7-d0ec958fcc4b\") " Sep 11 01:18:20.983390 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:20.983355 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cw64m\" (UniqueName: \"kubernetes.io/projected/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b-kube-api-access-cw64m\") pod \"9356eb13-4c12-4d62-b8f7-d0ec958fcc4b\" (UID: \"9356eb13-4c12-4d62-b8f7-d0ec958fcc4b\") " Sep 11 01:18:20.983641 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:20.983615 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9356eb13-4c12-4d62-b8f7-d0ec958fcc4b" (UID: "9356eb13-4c12-4d62-b8f7-d0ec958fcc4b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:18:20.985753 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:20.985727 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b-kube-api-access-cw64m" (OuterVolumeSpecName: "kube-api-access-cw64m") pod "9356eb13-4c12-4d62-b8f7-d0ec958fcc4b" (UID: "9356eb13-4c12-4d62-b8f7-d0ec958fcc4b"). InnerVolumeSpecName "kube-api-access-cw64m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:18:20.985864 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:20.985734 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9356eb13-4c12-4d62-b8f7-d0ec958fcc4b" (UID: "9356eb13-4c12-4d62-b8f7-d0ec958fcc4b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:18:20.985864 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:20.985770 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9356eb13-4c12-4d62-b8f7-d0ec958fcc4b" (UID: "9356eb13-4c12-4d62-b8f7-d0ec958fcc4b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:18:21.042613 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:21.042574 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:18:21.084398 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:21.084376 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b-metrics-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:21.084398 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:21.084397 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b-maas-api-tls\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:21.084526 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:21.084407 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw64m\" (UniqueName: \"kubernetes.io/projected/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b-kube-api-access-cw64m\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:21.084526 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:21.084416 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b-openshift-service-ca\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:21.309707 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:21.309633 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:18:21.532327 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:21.532295 2554 generic.go:356] "Generic (PLEG): container finished" podID="9356eb13-4c12-4d62-b8f7-d0ec958fcc4b" containerID="22ac1554a4f60f14b0dab08ff7676688baf2416f040acdd4a95c3bb99295cb49" exitCode=0 Sep 11 01:18:21.532495 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:21.532349 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2" Sep 11 01:18:21.532495 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:21.532379 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2" event={"ID":"9356eb13-4c12-4d62-b8f7-d0ec958fcc4b","Type":"ContainerDied","Data":"22ac1554a4f60f14b0dab08ff7676688baf2416f040acdd4a95c3bb99295cb49"} Sep 11 01:18:21.532495 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:21.532434 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2" event={"ID":"9356eb13-4c12-4d62-b8f7-d0ec958fcc4b","Type":"ContainerDied","Data":"6d0166049f365b4ad5654c808eb8cefe152afcdd233b241f92cebe76ffb0010d"} Sep 11 01:18:21.532495 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:21.532456 2554 scope.go:117] "RemoveContainer" containerID="22ac1554a4f60f14b0dab08ff7676688baf2416f040acdd4a95c3bb99295cb49" Sep 11 01:18:21.545069 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:21.545043 2554 scope.go:117] "RemoveContainer" containerID="22ac1554a4f60f14b0dab08ff7676688baf2416f040acdd4a95c3bb99295cb49" Sep 11 01:18:21.545420 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:18:21.545392 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22ac1554a4f60f14b0dab08ff7676688baf2416f040acdd4a95c3bb99295cb49\": container with ID starting with 22ac1554a4f60f14b0dab08ff7676688baf2416f040acdd4a95c3bb99295cb49 not found: ID does not exist" containerID="22ac1554a4f60f14b0dab08ff7676688baf2416f040acdd4a95c3bb99295cb49" Sep 11 01:18:21.545511 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:21.545430 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ac1554a4f60f14b0dab08ff7676688baf2416f040acdd4a95c3bb99295cb49"} err="failed to get container status \"22ac1554a4f60f14b0dab08ff7676688baf2416f040acdd4a95c3bb99295cb49\": rpc error: code = NotFound desc = could not find container \"22ac1554a4f60f14b0dab08ff7676688baf2416f040acdd4a95c3bb99295cb49\": container with ID starting with 22ac1554a4f60f14b0dab08ff7676688baf2416f040acdd4a95c3bb99295cb49 not found: ID does not exist" Sep 11 01:18:21.556702 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:21.556683 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2"] Sep 11 01:18:21.563119 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:21.563069 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-891e67-8485698886-nmbg2"] Sep 11 01:18:21.678128 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:21.678100 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9356eb13-4c12-4d62-b8f7-d0ec958fcc4b" path="/var/lib/kubelet/pods/9356eb13-4c12-4d62-b8f7-d0ec958fcc4b/volumes" Sep 11 01:18:26.290521 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:26.290488 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr"] Sep 11 01:18:26.339200 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:26.339170 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:18:40.872613 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:40.872567 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:18:40.973315 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:40.973282 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:18:41.024950 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.024920 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:18:41.309645 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.309612 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" podUID="ed80a8bf-9214-4bfc-80c8-896c66d5e5ff" containerName="main" containerID="cri-o://e7982ca5be61adb9a64096013af2ed84edc6f21fbf201448c08be8eb9934ad38" gracePeriod=45 Sep 11 01:18:41.559251 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.559226 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:18:41.640738 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.640653 2554 generic.go:356] "Generic (PLEG): container finished" podID="ed80a8bf-9214-4bfc-80c8-896c66d5e5ff" containerID="e7982ca5be61adb9a64096013af2ed84edc6f21fbf201448c08be8eb9934ad38" exitCode=0 Sep 11 01:18:41.640738 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.640704 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" Sep 11 01:18:41.640969 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.640735 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" event={"ID":"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff","Type":"ContainerDied","Data":"e7982ca5be61adb9a64096013af2ed84edc6f21fbf201448c08be8eb9934ad38"} Sep 11 01:18:41.640969 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.640772 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr" event={"ID":"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff","Type":"ContainerDied","Data":"dc3cf47de600c5404b9c6309087120a0417e45ba99c9bb72310d1f287e4f62bf"} Sep 11 01:18:41.640969 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.640793 2554 scope.go:117] "RemoveContainer" containerID="e7982ca5be61adb9a64096013af2ed84edc6f21fbf201448c08be8eb9934ad38" Sep 11 01:18:41.649319 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.649304 2554 scope.go:117] "RemoveContainer" containerID="e7982ca5be61adb9a64096013af2ed84edc6f21fbf201448c08be8eb9934ad38" Sep 11 01:18:41.649571 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:18:41.649553 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7982ca5be61adb9a64096013af2ed84edc6f21fbf201448c08be8eb9934ad38\": container with ID starting with e7982ca5be61adb9a64096013af2ed84edc6f21fbf201448c08be8eb9934ad38 not found: ID does not exist" containerID="e7982ca5be61adb9a64096013af2ed84edc6f21fbf201448c08be8eb9934ad38" Sep 11 01:18:41.649629 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.649587 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7982ca5be61adb9a64096013af2ed84edc6f21fbf201448c08be8eb9934ad38"} err="failed to get container status \"e7982ca5be61adb9a64096013af2ed84edc6f21fbf201448c08be8eb9934ad38\": rpc error: code = NotFound desc = could not find container \"e7982ca5be61adb9a64096013af2ed84edc6f21fbf201448c08be8eb9934ad38\": container with ID starting with e7982ca5be61adb9a64096013af2ed84edc6f21fbf201448c08be8eb9934ad38 not found: ID does not exist" Sep 11 01:18:41.653908 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.653891 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-tmp-dir\") pod \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\" (UID: \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\") " Sep 11 01:18:41.653975 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.653919 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-model-cache\") pod \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\" (UID: \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\") " Sep 11 01:18:41.653975 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.653964 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-tls-certs\") pod \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\" (UID: \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\") " Sep 11 01:18:41.654039 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.653992 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-dshm\") pod \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\" (UID: \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\") " Sep 11 01:18:41.654072 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.654045 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzh2f\" (UniqueName: \"kubernetes.io/projected/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-kube-api-access-pzh2f\") pod \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\" (UID: \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\") " Sep 11 01:18:41.654072 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.654064 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-home\") pod \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\" (UID: \"ed80a8bf-9214-4bfc-80c8-896c66d5e5ff\") " Sep 11 01:18:41.654447 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.654146 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ed80a8bf-9214-4bfc-80c8-896c66d5e5ff" (UID: "ed80a8bf-9214-4bfc-80c8-896c66d5e5ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:18:41.654447 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.654226 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-model-cache" (OuterVolumeSpecName: "model-cache") pod "ed80a8bf-9214-4bfc-80c8-896c66d5e5ff" (UID: "ed80a8bf-9214-4bfc-80c8-896c66d5e5ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:18:41.654447 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.654424 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-tmp-dir\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:41.654447 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.654440 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-model-cache\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:41.654447 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.654418 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-home" (OuterVolumeSpecName: "home") pod "ed80a8bf-9214-4bfc-80c8-896c66d5e5ff" (UID: "ed80a8bf-9214-4bfc-80c8-896c66d5e5ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:18:41.656131 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.656104 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-kube-api-access-pzh2f" (OuterVolumeSpecName: "kube-api-access-pzh2f") pod "ed80a8bf-9214-4bfc-80c8-896c66d5e5ff" (UID: "ed80a8bf-9214-4bfc-80c8-896c66d5e5ff"). InnerVolumeSpecName "kube-api-access-pzh2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:18:41.656220 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.656133 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-dshm" (OuterVolumeSpecName: "dshm") pod "ed80a8bf-9214-4bfc-80c8-896c66d5e5ff" (UID: "ed80a8bf-9214-4bfc-80c8-896c66d5e5ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:18:41.656259 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.656211 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ed80a8bf-9214-4bfc-80c8-896c66d5e5ff" (UID: "ed80a8bf-9214-4bfc-80c8-896c66d5e5ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:18:41.755000 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.754973 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-tls-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:41.755000 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.755000 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-dshm\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:41.755140 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.755016 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzh2f\" (UniqueName: \"kubernetes.io/projected/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-kube-api-access-pzh2f\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:41.755140 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.755030 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff-home\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:18:41.963578 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.963539 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr"] Sep 11 01:18:41.964933 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:41.964906 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-891e67/test-model-891e67-kserve-55bfc86ffc-nf9kr"] Sep 11 01:18:43.678524 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:43.678491 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed80a8bf-9214-4bfc-80c8-896c66d5e5ff" path="/var/lib/kubelet/pods/ed80a8bf-9214-4bfc-80c8-896c66d5e5ff/volumes" Sep 11 01:18:53.777945 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:18:53.773990 2554 scope.go:117] "RemoveContainer" containerID="54846cd21ced4d41e0716d9c4666ea479f4dd66293fe501fe1df734798fae951" Sep 11 01:19:00.740285 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:00.740252 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:02.649667 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:02.649635 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:03.537301 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:03.537261 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:12.326928 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:12.326888 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:15.745111 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:15.745074 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:15.976112 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:15.976081 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:16.084626 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:16.084543 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:18.011304 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:18.011267 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:18.112741 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:18.112708 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:18.267584 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:18.267503 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:18.830292 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:18.830259 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:19.144462 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:19.144365 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:19.318533 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:19.318499 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:21.839924 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:21.839879 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:21.927983 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:21.927947 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:22.489508 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:22.489476 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:22.564890 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:22.564836 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:23.149652 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:23.149625 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:23.251565 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:23.251528 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:23.530788 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:23.530755 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:23.630463 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:23.630428 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:23.733148 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:23.733117 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:27.551210 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:27.551179 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:27.619188 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:27.619157 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:32.212149 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:32.212108 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:46.854261 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:46.854225 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:47.210049 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:47.210016 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:47.512151 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:47.512065 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:47.611684 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:47.611649 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:50.673396 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:50.673363 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:50.764760 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:50.764719 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:51.830437 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:51.830402 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:51.902134 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:51.902106 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:52.097317 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:52.097244 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:52.397747 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:52.397653 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:19:52.493675 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:19:52.493644 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:14.339826 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:14.339790 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:15.707733 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:15.707699 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:23.977256 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:23.977217 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:24.503606 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:24.503570 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6"] Sep 11 01:20:24.503865 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:24.503819 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6" podUID="39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb" containerName="maas-api" containerID="cri-o://58b6ed35be2753994476f3967ee8917c4bc05008b3a8cd3c727a971ed59a888a" gracePeriod=30 Sep 11 01:20:24.533281 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:24.533254 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:24.634396 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:24.634306 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:24.720230 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:24.720185 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:24.754535 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:24.754486 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6" Sep 11 01:20:24.783196 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:24.783165 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:24.812984 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:24.812963 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb-metrics-certs\") pod \"39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb\" (UID: \"39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb\") " Sep 11 01:20:24.813104 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:24.812995 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb-maas-api-tls\") pod \"39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb\" (UID: \"39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb\") " Sep 11 01:20:24.813104 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:24.813017 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb-openshift-service-ca\") pod \"39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb\" (UID: \"39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb\") " Sep 11 01:20:24.813104 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:24.813054 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxjkt\" (UniqueName: \"kubernetes.io/projected/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb-kube-api-access-nxjkt\") pod \"39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb\" (UID: \"39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb\") " Sep 11 01:20:24.813333 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:24.813311 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb" (UID: "39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:20:24.815077 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:24.815057 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb-kube-api-access-nxjkt" (OuterVolumeSpecName: "kube-api-access-nxjkt") pod "39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb" (UID: "39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb"). InnerVolumeSpecName "kube-api-access-nxjkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:20:24.815185 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:24.815136 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb" (UID: "39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:20:24.815235 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:24.815195 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb" (UID: "39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:20:24.914160 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:24.914133 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxjkt\" (UniqueName: \"kubernetes.io/projected/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb-kube-api-access-nxjkt\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:20:24.914160 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:24.914157 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb-metrics-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:20:24.914300 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:24.914166 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb-maas-api-tls\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:20:24.914300 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:24.914177 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb-openshift-service-ca\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:20:25.166511 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:25.166430 2554 generic.go:356] "Generic (PLEG): container finished" podID="39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb" containerID="58b6ed35be2753994476f3967ee8917c4bc05008b3a8cd3c727a971ed59a888a" exitCode=0 Sep 11 01:20:25.166511 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:25.166474 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6" Sep 11 01:20:25.167023 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:25.166513 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6" event={"ID":"39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb","Type":"ContainerDied","Data":"58b6ed35be2753994476f3967ee8917c4bc05008b3a8cd3c727a971ed59a888a"} Sep 11 01:20:25.167023 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:25.166571 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6" event={"ID":"39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb","Type":"ContainerDied","Data":"db2c50c1a6193175c8464d86947ebd8536de33e10dcc1d28dddd68f6920ff35c"} Sep 11 01:20:25.167023 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:25.166603 2554 scope.go:117] "RemoveContainer" containerID="58b6ed35be2753994476f3967ee8917c4bc05008b3a8cd3c727a971ed59a888a" Sep 11 01:20:25.174249 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:25.174176 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:25.178927 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:25.178689 2554 scope.go:117] "RemoveContainer" containerID="58b6ed35be2753994476f3967ee8917c4bc05008b3a8cd3c727a971ed59a888a" Sep 11 01:20:25.185796 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:20:25.185765 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b6ed35be2753994476f3967ee8917c4bc05008b3a8cd3c727a971ed59a888a\": container with ID starting with 58b6ed35be2753994476f3967ee8917c4bc05008b3a8cd3c727a971ed59a888a not found: ID does not exist" containerID="58b6ed35be2753994476f3967ee8917c4bc05008b3a8cd3c727a971ed59a888a" Sep 11 01:20:25.185925 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:25.185804 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b6ed35be2753994476f3967ee8917c4bc05008b3a8cd3c727a971ed59a888a"} err="failed to get container status \"58b6ed35be2753994476f3967ee8917c4bc05008b3a8cd3c727a971ed59a888a\": rpc error: code = NotFound desc = could not find container \"58b6ed35be2753994476f3967ee8917c4bc05008b3a8cd3c727a971ed59a888a\": container with ID starting with 58b6ed35be2753994476f3967ee8917c4bc05008b3a8cd3c727a971ed59a888a not found: ID does not exist" Sep 11 01:20:25.189297 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:25.189267 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6"] Sep 11 01:20:25.192703 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:25.192683 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-194eb64c-65f7d67d79-thwq6"] Sep 11 01:20:25.678796 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:25.678767 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb" path="/var/lib/kubelet/pods/39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb/volumes" Sep 11 01:20:30.802138 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:30.802101 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:30.900183 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:30.900153 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:31.104570 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:31.104487 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:32.598220 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:32.598189 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:32.697139 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:32.697099 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:32.796104 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:32.796077 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:33.468252 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:33.468221 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:33.576790 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:33.576753 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:33.667060 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:33.667028 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:35.097189 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:35.097150 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:40.902080 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:40.902041 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:41.003771 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:41.003739 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:41.697048 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:41.696990 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:45.609295 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:45.609260 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:47.420387 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:47.420356 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:47.519708 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:47.519654 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:47.617174 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:47.617135 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:58.153919 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:58.153884 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:20:58.384519 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:20:58.384476 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:00.412861 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:00.412816 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:18.496890 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.496853 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct"] Sep 11 01:21:18.497403 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.497271 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9356eb13-4c12-4d62-b8f7-d0ec958fcc4b" containerName="maas-api" Sep 11 01:21:18.497403 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.497289 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9356eb13-4c12-4d62-b8f7-d0ec958fcc4b" containerName="maas-api" Sep 11 01:21:18.497403 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.497302 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb" containerName="maas-api" Sep 11 01:21:18.497403 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.497310 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb" containerName="maas-api" Sep 11 01:21:18.497403 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.497321 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2093b37-1516-4fdf-a834-f6c039b0b05c" containerName="main" Sep 11 01:21:18.497403 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.497331 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2093b37-1516-4fdf-a834-f6c039b0b05c" containerName="main" Sep 11 01:21:18.497403 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.497343 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaa119bd-3d65-4667-b2f3-50e7ac3dd00f" containerName="maas-api" Sep 11 01:21:18.497403 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.497353 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaa119bd-3d65-4667-b2f3-50e7ac3dd00f" containerName="maas-api" Sep 11 01:21:18.497403 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.497369 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2738da06-3225-42c9-b77a-86e864a9babd" containerName="main" Sep 11 01:21:18.497403 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.497377 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2738da06-3225-42c9-b77a-86e864a9babd" containerName="main" Sep 11 01:21:18.497403 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.497385 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed80a8bf-9214-4bfc-80c8-896c66d5e5ff" containerName="main" Sep 11 01:21:18.497403 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.497393 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed80a8bf-9214-4bfc-80c8-896c66d5e5ff" containerName="main" Sep 11 01:21:18.498074 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.497539 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaa119bd-3d65-4667-b2f3-50e7ac3dd00f" containerName="maas-api" Sep 11 01:21:18.498074 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.497553 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed80a8bf-9214-4bfc-80c8-896c66d5e5ff" containerName="main" Sep 11 01:21:18.498074 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.497562 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2093b37-1516-4fdf-a834-f6c039b0b05c" containerName="main" Sep 11 01:21:18.498074 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.497571 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9356eb13-4c12-4d62-b8f7-d0ec958fcc4b" containerName="maas-api" Sep 11 01:21:18.498074 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.497581 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="39d50f6e-ebe6-4b8a-9ff7-46a1f2c9ebcb" containerName="maas-api" Sep 11 01:21:18.498074 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.497590 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2738da06-3225-42c9-b77a-86e864a9babd" containerName="main" Sep 11 01:21:18.502198 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.502175 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:21:18.504423 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.504400 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2482ac\"/\"kube-root-ca.crt\"" Sep 11 01:21:18.505518 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.505494 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2482ac\"/\"auth-test-model-2482ac-kserve-self-signed-certs\"" Sep 11 01:21:18.505640 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.505526 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2482ac\"/\"default-dockercfg-w9chz\"" Sep 11 01:21:18.505640 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.505524 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2482ac\"/\"openshift-service-ca.crt\"" Sep 11 01:21:18.508494 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.508467 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct"] Sep 11 01:21:18.527280 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.527252 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:18.578784 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.578759 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-dshm\") pod \"auth-test-model-2482ac-kserve-84ccf6b46c-w5fct\" (UID: \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:21:18.578902 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.578788 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-model-cache\") pod \"auth-test-model-2482ac-kserve-84ccf6b46c-w5fct\" (UID: \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:21:18.578902 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.578811 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9clt\" (UniqueName: \"kubernetes.io/projected/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-kube-api-access-v9clt\") pod \"auth-test-model-2482ac-kserve-84ccf6b46c-w5fct\" (UID: \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:21:18.579001 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.578928 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-tmp-dir\") pod \"auth-test-model-2482ac-kserve-84ccf6b46c-w5fct\" (UID: \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:21:18.579054 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.579011 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-home\") pod \"auth-test-model-2482ac-kserve-84ccf6b46c-w5fct\" (UID: \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:21:18.579106 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.579066 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-tls-certs\") pod \"auth-test-model-2482ac-kserve-84ccf6b46c-w5fct\" (UID: \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:21:18.680311 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.680285 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-home\") pod \"auth-test-model-2482ac-kserve-84ccf6b46c-w5fct\" (UID: \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:21:18.680408 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.680322 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-tls-certs\") pod \"auth-test-model-2482ac-kserve-84ccf6b46c-w5fct\" (UID: \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:21:18.680408 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.680353 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-dshm\") pod \"auth-test-model-2482ac-kserve-84ccf6b46c-w5fct\" (UID: \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:21:18.680408 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.680385 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-model-cache\") pod \"auth-test-model-2482ac-kserve-84ccf6b46c-w5fct\" (UID: \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:21:18.680516 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.680415 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9clt\" (UniqueName: \"kubernetes.io/projected/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-kube-api-access-v9clt\") pod \"auth-test-model-2482ac-kserve-84ccf6b46c-w5fct\" (UID: \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:21:18.680516 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.680493 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-tmp-dir\") pod \"auth-test-model-2482ac-kserve-84ccf6b46c-w5fct\" (UID: \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:21:18.680745 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.680724 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-home\") pod \"auth-test-model-2482ac-kserve-84ccf6b46c-w5fct\" (UID: \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:21:18.680821 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.680754 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-model-cache\") pod \"auth-test-model-2482ac-kserve-84ccf6b46c-w5fct\" (UID: \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:21:18.680916 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.680855 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-tmp-dir\") pod \"auth-test-model-2482ac-kserve-84ccf6b46c-w5fct\" (UID: \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:21:18.682661 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.682643 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-dshm\") pod \"auth-test-model-2482ac-kserve-84ccf6b46c-w5fct\" (UID: \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:21:18.682809 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.682792 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-tls-certs\") pod \"auth-test-model-2482ac-kserve-84ccf6b46c-w5fct\" (UID: \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:21:18.687631 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.687610 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9clt\" (UniqueName: \"kubernetes.io/projected/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-kube-api-access-v9clt\") pod \"auth-test-model-2482ac-kserve-84ccf6b46c-w5fct\" (UID: \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:21:18.813582 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:18.813527 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:21:19.088768 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:19.088700 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct"] Sep 11 01:21:19.092100 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:21:19.092070 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2c1c1cd_8a80_4738_ac30_604ee1744a2f.slice/crio-1733a932c47ecd2b5544d1c5f6b6fd32cbdd6bf035f7af0d8af60f472c7e9995 WatchSource:0}: Error finding container 1733a932c47ecd2b5544d1c5f6b6fd32cbdd6bf035f7af0d8af60f472c7e9995: Status 404 returned error can't find the container with id 1733a932c47ecd2b5544d1c5f6b6fd32cbdd6bf035f7af0d8af60f472c7e9995 Sep 11 01:21:19.385409 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:19.385324 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:19.444076 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:19.444042 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" event={"ID":"a2c1c1cd-8a80-4738-ac30-604ee1744a2f","Type":"ContainerStarted","Data":"df0b406e55d95a0054815018d1024ee31101f0bb5e3d82f31b4229250c522745"} Sep 11 01:21:19.444252 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:19.444092 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" event={"ID":"a2c1c1cd-8a80-4738-ac30-604ee1744a2f","Type":"ContainerStarted","Data":"1733a932c47ecd2b5544d1c5f6b6fd32cbdd6bf035f7af0d8af60f472c7e9995"} Sep 11 01:21:19.484708 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:19.484662 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" podStartSLOduration=1.484646503 podStartE2EDuration="1.484646503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:21:19.479782164 +0000 UTC m=+1406.303235673" watchObservedRunningTime="2026-09-11 01:21:19.484646503 +0000 UTC m=+1406.308100053" Sep 11 01:21:24.792288 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:24.792221 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:25.157118 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:25.157037 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:25.323206 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:25.323167 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:25.521985 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:25.521952 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:25.839659 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:25.839580 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:26.078552 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:26.078518 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:28.249000 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:28.248959 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:28.703226 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:28.703193 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:28.804122 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:28.804091 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:28.814504 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:28.814481 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:21:28.814628 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:28.814519 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:21:28.827067 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:28.827047 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:21:29.507424 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:29.507394 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:21:29.842536 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:29.841766 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9"] Sep 11 01:21:29.845988 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:29.845965 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9" Sep 11 01:21:29.849058 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:29.848835 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f1f91d-serving-cert\"" Sep 11 01:21:29.864146 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:29.864121 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9"] Sep 11 01:21:29.905437 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:29.905387 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:29.981267 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:29.981231 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b36b03a6-107a-461b-8a93-013521b46c1f-metrics-certs\") pod \"maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9\" (UID: \"b36b03a6-107a-461b-8a93-013521b46c1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9" Sep 11 01:21:29.981411 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:29.981285 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b36b03a6-107a-461b-8a93-013521b46c1f-maas-api-tls\") pod \"maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9\" (UID: \"b36b03a6-107a-461b-8a93-013521b46c1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9" Sep 11 01:21:29.981460 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:29.981420 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36b03a6-107a-461b-8a93-013521b46c1f-openshift-service-ca\") pod \"maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9\" (UID: \"b36b03a6-107a-461b-8a93-013521b46c1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9" Sep 11 01:21:29.981514 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:29.981462 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c76p2\" (UniqueName: \"kubernetes.io/projected/b36b03a6-107a-461b-8a93-013521b46c1f-kube-api-access-c76p2\") pod \"maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9\" (UID: \"b36b03a6-107a-461b-8a93-013521b46c1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9" Sep 11 01:21:30.072395 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:30.072362 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:30.082907 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:30.082885 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b36b03a6-107a-461b-8a93-013521b46c1f-metrics-certs\") pod \"maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9\" (UID: \"b36b03a6-107a-461b-8a93-013521b46c1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9" Sep 11 01:21:30.083008 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:30.082918 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b36b03a6-107a-461b-8a93-013521b46c1f-maas-api-tls\") pod \"maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9\" (UID: \"b36b03a6-107a-461b-8a93-013521b46c1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9" Sep 11 01:21:30.083008 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:30.082988 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36b03a6-107a-461b-8a93-013521b46c1f-openshift-service-ca\") pod \"maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9\" (UID: \"b36b03a6-107a-461b-8a93-013521b46c1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9" Sep 11 01:21:30.083093 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:30.083009 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c76p2\" (UniqueName: \"kubernetes.io/projected/b36b03a6-107a-461b-8a93-013521b46c1f-kube-api-access-c76p2\") pod \"maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9\" (UID: \"b36b03a6-107a-461b-8a93-013521b46c1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9" Sep 11 01:21:30.083646 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:30.083624 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36b03a6-107a-461b-8a93-013521b46c1f-openshift-service-ca\") pod \"maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9\" (UID: \"b36b03a6-107a-461b-8a93-013521b46c1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9" Sep 11 01:21:30.085348 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:30.085331 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b36b03a6-107a-461b-8a93-013521b46c1f-metrics-certs\") pod \"maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9\" (UID: \"b36b03a6-107a-461b-8a93-013521b46c1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9" Sep 11 01:21:30.085427 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:30.085408 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b36b03a6-107a-461b-8a93-013521b46c1f-maas-api-tls\") pod \"maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9\" (UID: \"b36b03a6-107a-461b-8a93-013521b46c1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9" Sep 11 01:21:30.092938 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:30.092873 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c76p2\" (UniqueName: \"kubernetes.io/projected/b36b03a6-107a-461b-8a93-013521b46c1f-kube-api-access-c76p2\") pod \"maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9\" (UID: \"b36b03a6-107a-461b-8a93-013521b46c1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9" Sep 11 01:21:30.157387 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:30.157365 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9" Sep 11 01:21:30.429168 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:30.429136 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9"] Sep 11 01:21:30.430932 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:21:30.430900 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36b03a6_107a_461b_8a93_013521b46c1f.slice/crio-ed597d997fde5c6b1ba774de8f98f08117fc5c1c8e2c3ec4232889034ffeb5ea WatchSource:0}: Error finding container ed597d997fde5c6b1ba774de8f98f08117fc5c1c8e2c3ec4232889034ffeb5ea: Status 404 returned error can't find the container with id ed597d997fde5c6b1ba774de8f98f08117fc5c1c8e2c3ec4232889034ffeb5ea Sep 11 01:21:30.432972 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:30.432712 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 01:21:30.474450 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:30.474419 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:30.504832 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:30.504803 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9" event={"ID":"b36b03a6-107a-461b-8a93-013521b46c1f","Type":"ContainerStarted","Data":"ed597d997fde5c6b1ba774de8f98f08117fc5c1c8e2c3ec4232889034ffeb5ea"} Sep 11 01:21:30.571243 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:30.571196 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:31.511739 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:31.511706 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9" event={"ID":"b36b03a6-107a-461b-8a93-013521b46c1f","Type":"ContainerStarted","Data":"668129cd484b5ce34bd9bf78f438ab74e2297ec356b0fe45634463faa2c0e18c"} Sep 11 01:21:31.511952 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:31.511818 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9" Sep 11 01:21:31.532918 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:31.532873 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9" podStartSLOduration=2.076138731 podStartE2EDuration="2.532833399s" totalImagesPullingTime="456.694668ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:21:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:21:31.531913734 +0000 UTC m=+1418.355367243" watchObservedRunningTime="2026-09-11 01:21:31.532833399 +0000 UTC m=+1418.356286914" Sep 11 01:21:37.522359 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:37.522322 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9" Sep 11 01:21:53.617792 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:53.617692 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:54.129406 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:54.129369 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:54.427296 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:54.427264 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:54.554535 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:54.554505 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:57.762557 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:57.762524 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:57.862457 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:57.862422 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:58.893168 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:58.893134 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj"] Sep 11 01:21:58.896806 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:58.896784 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" Sep 11 01:21:58.898963 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:58.898940 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-816605-serving-cert\"" Sep 11 01:21:58.906625 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:58.906603 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj"] Sep 11 01:21:58.924885 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:58.924861 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:58.926219 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:58.926193 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npp4l\" (UniqueName: \"kubernetes.io/projected/ccc8fbc2-92a6-44b3-84b0-f064114cc151-kube-api-access-npp4l\") pod \"maas-api-e2e-api-b-816605-99bf745ff-lbmjj\" (UID: \"ccc8fbc2-92a6-44b3-84b0-f064114cc151\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" Sep 11 01:21:58.926328 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:58.926258 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc8fbc2-92a6-44b3-84b0-f064114cc151-openshift-service-ca\") pod \"maas-api-e2e-api-b-816605-99bf745ff-lbmjj\" (UID: \"ccc8fbc2-92a6-44b3-84b0-f064114cc151\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" Sep 11 01:21:58.926328 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:58.926306 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccc8fbc2-92a6-44b3-84b0-f064114cc151-metrics-certs\") pod \"maas-api-e2e-api-b-816605-99bf745ff-lbmjj\" (UID: \"ccc8fbc2-92a6-44b3-84b0-f064114cc151\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" Sep 11 01:21:58.926447 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:58.926338 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccc8fbc2-92a6-44b3-84b0-f064114cc151-maas-api-tls\") pod \"maas-api-e2e-api-b-816605-99bf745ff-lbmjj\" (UID: \"ccc8fbc2-92a6-44b3-84b0-f064114cc151\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" Sep 11 01:21:58.983538 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:58.983513 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:59.026738 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:59.026714 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npp4l\" (UniqueName: \"kubernetes.io/projected/ccc8fbc2-92a6-44b3-84b0-f064114cc151-kube-api-access-npp4l\") pod \"maas-api-e2e-api-b-816605-99bf745ff-lbmjj\" (UID: \"ccc8fbc2-92a6-44b3-84b0-f064114cc151\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" Sep 11 01:21:59.026826 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:59.026758 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc8fbc2-92a6-44b3-84b0-f064114cc151-openshift-service-ca\") pod \"maas-api-e2e-api-b-816605-99bf745ff-lbmjj\" (UID: \"ccc8fbc2-92a6-44b3-84b0-f064114cc151\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" Sep 11 01:21:59.026826 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:59.026787 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccc8fbc2-92a6-44b3-84b0-f064114cc151-metrics-certs\") pod \"maas-api-e2e-api-b-816605-99bf745ff-lbmjj\" (UID: \"ccc8fbc2-92a6-44b3-84b0-f064114cc151\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" Sep 11 01:21:59.026826 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:59.026809 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccc8fbc2-92a6-44b3-84b0-f064114cc151-maas-api-tls\") pod \"maas-api-e2e-api-b-816605-99bf745ff-lbmjj\" (UID: \"ccc8fbc2-92a6-44b3-84b0-f064114cc151\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" Sep 11 01:21:59.026979 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:21:59.026949 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-816605-serving-cert: secret "maas-api-e2e-api-b-816605-serving-cert" not found Sep 11 01:21:59.027024 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:21:59.027005 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccc8fbc2-92a6-44b3-84b0-f064114cc151-maas-api-tls podName:ccc8fbc2-92a6-44b3-84b0-f064114cc151 nodeName:}" failed. No retries permitted until 2026-09-11 01:21:59.526988084 +0000 UTC m=+1446.350441586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ccc8fbc2-92a6-44b3-84b0-f064114cc151-maas-api-tls") pod "maas-api-e2e-api-b-816605-99bf745ff-lbmjj" (UID: "ccc8fbc2-92a6-44b3-84b0-f064114cc151") : secret "maas-api-e2e-api-b-816605-serving-cert" not found Sep 11 01:21:59.027537 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:59.027516 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc8fbc2-92a6-44b3-84b0-f064114cc151-openshift-service-ca\") pod \"maas-api-e2e-api-b-816605-99bf745ff-lbmjj\" (UID: \"ccc8fbc2-92a6-44b3-84b0-f064114cc151\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" Sep 11 01:21:59.029206 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:59.029185 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccc8fbc2-92a6-44b3-84b0-f064114cc151-metrics-certs\") pod \"maas-api-e2e-api-b-816605-99bf745ff-lbmjj\" (UID: \"ccc8fbc2-92a6-44b3-84b0-f064114cc151\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" Sep 11 01:21:59.034506 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:59.034481 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npp4l\" (UniqueName: \"kubernetes.io/projected/ccc8fbc2-92a6-44b3-84b0-f064114cc151-kube-api-access-npp4l\") pod \"maas-api-e2e-api-b-816605-99bf745ff-lbmjj\" (UID: \"ccc8fbc2-92a6-44b3-84b0-f064114cc151\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" Sep 11 01:21:59.161433 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:59.161358 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:59.404485 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:59.404455 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:59.507409 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:59.507371 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:21:59.532119 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:59.532093 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccc8fbc2-92a6-44b3-84b0-f064114cc151-maas-api-tls\") pod \"maas-api-e2e-api-b-816605-99bf745ff-lbmjj\" (UID: \"ccc8fbc2-92a6-44b3-84b0-f064114cc151\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" Sep 11 01:21:59.534442 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:59.534420 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccc8fbc2-92a6-44b3-84b0-f064114cc151-maas-api-tls\") pod \"maas-api-e2e-api-b-816605-99bf745ff-lbmjj\" (UID: \"ccc8fbc2-92a6-44b3-84b0-f064114cc151\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" Sep 11 01:21:59.809739 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:21:59.809656 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" Sep 11 01:22:00.085317 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:00.085291 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj"] Sep 11 01:22:00.087329 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:22:00.087294 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc8fbc2_92a6_44b3_84b0_f064114cc151.slice/crio-6aa647ebce9a0901696e2e67f7c49cc09babb8cceee29076f37f34a89a10d2a0 WatchSource:0}: Error finding container 6aa647ebce9a0901696e2e67f7c49cc09babb8cceee29076f37f34a89a10d2a0: Status 404 returned error can't find the container with id 6aa647ebce9a0901696e2e67f7c49cc09babb8cceee29076f37f34a89a10d2a0 Sep 11 01:22:00.662766 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:00.662727 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" event={"ID":"ccc8fbc2-92a6-44b3-84b0-f064114cc151","Type":"ContainerStarted","Data":"fad06d0c65b450b2c72eb50113bb8d7baacf01127b9ba20115fa27b0d423b4cc"} Sep 11 01:22:00.662958 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:00.662785 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" event={"ID":"ccc8fbc2-92a6-44b3-84b0-f064114cc151","Type":"ContainerStarted","Data":"6aa647ebce9a0901696e2e67f7c49cc09babb8cceee29076f37f34a89a10d2a0"} Sep 11 01:22:00.662958 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:00.662825 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" Sep 11 01:22:00.679565 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:00.679520 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" podStartSLOduration=2.2797090669999998 podStartE2EDuration="2.679505843s" totalImagesPullingTime="399.796776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:21:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:22:00.678202816 +0000 UTC m=+1447.501656330" watchObservedRunningTime="2026-09-11 01:22:00.679505843 +0000 UTC m=+1447.502959353" Sep 11 01:22:06.672773 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:06.672742 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" Sep 11 01:22:10.257186 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:10.257147 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:10.727055 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:10.727021 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:13.509572 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.509538 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn"] Sep 11 01:22:13.514404 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.514386 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:22:13.518046 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.518022 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9433b9\"/\"openshift-service-ca.crt\"" Sep 11 01:22:13.518046 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.518034 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9433b9\"/\"kube-root-ca.crt\"" Sep 11 01:22:13.518046 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.518027 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9433b9\"/\"auth-test-model-9433b9-kserve-self-signed-certs\"" Sep 11 01:22:13.518294 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.518079 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9433b9\"/\"default-dockercfg-9w2hq\"" Sep 11 01:22:13.523621 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.523598 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn"] Sep 11 01:22:13.555919 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.555891 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/983daa40-2b81-4a89-9e72-a719b5d93eb6-tls-certs\") pod \"auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn\" (UID: \"983daa40-2b81-4a89-9e72-a719b5d93eb6\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:22:13.556020 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.555954 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983daa40-2b81-4a89-9e72-a719b5d93eb6-tmp-dir\") pod \"auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn\" (UID: \"983daa40-2b81-4a89-9e72-a719b5d93eb6\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:22:13.556020 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.556001 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/983daa40-2b81-4a89-9e72-a719b5d93eb6-dshm\") pod \"auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn\" (UID: \"983daa40-2b81-4a89-9e72-a719b5d93eb6\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:22:13.556121 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.556099 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klkjn\" (UniqueName: \"kubernetes.io/projected/983daa40-2b81-4a89-9e72-a719b5d93eb6-kube-api-access-klkjn\") pod \"auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn\" (UID: \"983daa40-2b81-4a89-9e72-a719b5d93eb6\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:22:13.556158 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.556150 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/983daa40-2b81-4a89-9e72-a719b5d93eb6-model-cache\") pod \"auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn\" (UID: \"983daa40-2b81-4a89-9e72-a719b5d93eb6\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:22:13.556193 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.556180 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/983daa40-2b81-4a89-9e72-a719b5d93eb6-home\") pod \"auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn\" (UID: \"983daa40-2b81-4a89-9e72-a719b5d93eb6\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:22:13.609748 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.609721 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:13.657129 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.657103 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983daa40-2b81-4a89-9e72-a719b5d93eb6-tmp-dir\") pod \"auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn\" (UID: \"983daa40-2b81-4a89-9e72-a719b5d93eb6\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:22:13.657263 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.657165 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/983daa40-2b81-4a89-9e72-a719b5d93eb6-dshm\") pod \"auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn\" (UID: \"983daa40-2b81-4a89-9e72-a719b5d93eb6\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:22:13.657263 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.657243 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klkjn\" (UniqueName: \"kubernetes.io/projected/983daa40-2b81-4a89-9e72-a719b5d93eb6-kube-api-access-klkjn\") pod \"auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn\" (UID: \"983daa40-2b81-4a89-9e72-a719b5d93eb6\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:22:13.657394 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.657290 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/983daa40-2b81-4a89-9e72-a719b5d93eb6-model-cache\") pod \"auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn\" (UID: \"983daa40-2b81-4a89-9e72-a719b5d93eb6\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:22:13.657394 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.657333 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/983daa40-2b81-4a89-9e72-a719b5d93eb6-home\") pod \"auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn\" (UID: \"983daa40-2b81-4a89-9e72-a719b5d93eb6\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:22:13.657394 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.657362 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/983daa40-2b81-4a89-9e72-a719b5d93eb6-tls-certs\") pod \"auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn\" (UID: \"983daa40-2b81-4a89-9e72-a719b5d93eb6\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:22:13.657552 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.657531 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983daa40-2b81-4a89-9e72-a719b5d93eb6-tmp-dir\") pod \"auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn\" (UID: \"983daa40-2b81-4a89-9e72-a719b5d93eb6\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:22:13.657860 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.657807 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/983daa40-2b81-4a89-9e72-a719b5d93eb6-model-cache\") pod \"auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn\" (UID: \"983daa40-2b81-4a89-9e72-a719b5d93eb6\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:22:13.657980 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.657933 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/983daa40-2b81-4a89-9e72-a719b5d93eb6-home\") pod \"auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn\" (UID: \"983daa40-2b81-4a89-9e72-a719b5d93eb6\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:22:13.659509 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.659489 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/983daa40-2b81-4a89-9e72-a719b5d93eb6-dshm\") pod \"auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn\" (UID: \"983daa40-2b81-4a89-9e72-a719b5d93eb6\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:22:13.659803 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.659783 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/983daa40-2b81-4a89-9e72-a719b5d93eb6-tls-certs\") pod \"auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn\" (UID: \"983daa40-2b81-4a89-9e72-a719b5d93eb6\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:22:13.669114 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.669096 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klkjn\" (UniqueName: \"kubernetes.io/projected/983daa40-2b81-4a89-9e72-a719b5d93eb6-kube-api-access-klkjn\") pod \"auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn\" (UID: \"983daa40-2b81-4a89-9e72-a719b5d93eb6\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:22:13.824933 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:13.824860 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:22:14.103002 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:14.102979 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn"] Sep 11 01:22:14.104831 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:22:14.104806 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod983daa40_2b81_4a89_9e72_a719b5d93eb6.slice/crio-1a539893d63e41b75e37bd404ba71cd96375e36ab4c91aea694e880094fbfb30 WatchSource:0}: Error finding container 1a539893d63e41b75e37bd404ba71cd96375e36ab4c91aea694e880094fbfb30: Status 404 returned error can't find the container with id 1a539893d63e41b75e37bd404ba71cd96375e36ab4c91aea694e880094fbfb30 Sep 11 01:22:14.733525 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:14.733489 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" event={"ID":"983daa40-2b81-4a89-9e72-a719b5d93eb6","Type":"ContainerStarted","Data":"cc6704a62d0c36b0951498568b555908df2e04f12f7ddde943ddd9dfcd366d86"} Sep 11 01:22:14.733918 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:14.733535 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" event={"ID":"983daa40-2b81-4a89-9e72-a719b5d93eb6","Type":"ContainerStarted","Data":"1a539893d63e41b75e37bd404ba71cd96375e36ab4c91aea694e880094fbfb30"} Sep 11 01:22:14.751301 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:14.751258 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" podStartSLOduration=1.751243221 podStartE2EDuration="1.751243221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:22:14.749412104 +0000 UTC m=+1461.572865618" watchObservedRunningTime="2026-09-11 01:22:14.751243221 +0000 UTC m=+1461.574696763" Sep 11 01:22:15.229559 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:15.229526 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r"] Sep 11 01:22:15.261458 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:15.261434 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv"] Sep 11 01:22:15.271556 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:15.271517 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh"] Sep 11 01:22:15.326492 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:15.326463 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:19.784309 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:19.784269 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:22.768021 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:22.767989 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:23.825719 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:23.825676 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:22:23.825719 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:23.825725 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:22:23.838897 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:23.838870 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:22:24.795780 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:24.795753 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:22:25.934467 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:25.934428 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:26.484883 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.484838 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk"] Sep 11 01:22:26.485161 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.485115 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" podUID="07f53eed-c8ba-42ee-ac68-f1d1cd2594db" containerName="maas-api" containerID="cri-o://8996366c5eb110a5110eb1bb69a4bdc986c1b8539bd365fd3430271c17860e59" gracePeriod=30 Sep 11 01:22:26.516031 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.516007 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:26.592459 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.592424 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:26.692550 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.692523 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:26.740801 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.740746 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" Sep 11 01:22:26.750791 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.750763 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:26.796267 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.796236 2554 generic.go:356] "Generic (PLEG): container finished" podID="07f53eed-c8ba-42ee-ac68-f1d1cd2594db" containerID="8996366c5eb110a5110eb1bb69a4bdc986c1b8539bd365fd3430271c17860e59" exitCode=0 Sep 11 01:22:26.796386 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.796279 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" Sep 11 01:22:26.796386 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.796316 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" event={"ID":"07f53eed-c8ba-42ee-ac68-f1d1cd2594db","Type":"ContainerDied","Data":"8996366c5eb110a5110eb1bb69a4bdc986c1b8539bd365fd3430271c17860e59"} Sep 11 01:22:26.796386 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.796368 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk" event={"ID":"07f53eed-c8ba-42ee-ac68-f1d1cd2594db","Type":"ContainerDied","Data":"543a4a6407b17214c7c23bd63de77e665fea41e18cdb68f999f0f4eefa3922af"} Sep 11 01:22:26.796500 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.796390 2554 scope.go:117] "RemoveContainer" containerID="8996366c5eb110a5110eb1bb69a4bdc986c1b8539bd365fd3430271c17860e59" Sep 11 01:22:26.805992 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.805971 2554 scope.go:117] "RemoveContainer" containerID="8996366c5eb110a5110eb1bb69a4bdc986c1b8539bd365fd3430271c17860e59" Sep 11 01:22:26.806255 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:22:26.806235 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8996366c5eb110a5110eb1bb69a4bdc986c1b8539bd365fd3430271c17860e59\": container with ID starting with 8996366c5eb110a5110eb1bb69a4bdc986c1b8539bd365fd3430271c17860e59 not found: ID does not exist" containerID="8996366c5eb110a5110eb1bb69a4bdc986c1b8539bd365fd3430271c17860e59" Sep 11 01:22:26.806309 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.806262 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8996366c5eb110a5110eb1bb69a4bdc986c1b8539bd365fd3430271c17860e59"} err="failed to get container status \"8996366c5eb110a5110eb1bb69a4bdc986c1b8539bd365fd3430271c17860e59\": rpc error: code = NotFound desc = could not find container \"8996366c5eb110a5110eb1bb69a4bdc986c1b8539bd365fd3430271c17860e59\": container with ID starting with 8996366c5eb110a5110eb1bb69a4bdc986c1b8539bd365fd3430271c17860e59 not found: ID does not exist" Sep 11 01:22:26.887202 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.887177 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-openshift-service-ca\") pod \"07f53eed-c8ba-42ee-ac68-f1d1cd2594db\" (UID: \"07f53eed-c8ba-42ee-ac68-f1d1cd2594db\") " Sep 11 01:22:26.887345 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.887233 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxc6p\" (UniqueName: \"kubernetes.io/projected/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-kube-api-access-hxc6p\") pod \"07f53eed-c8ba-42ee-ac68-f1d1cd2594db\" (UID: \"07f53eed-c8ba-42ee-ac68-f1d1cd2594db\") " Sep 11 01:22:26.887345 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.887287 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-metrics-certs\") pod \"07f53eed-c8ba-42ee-ac68-f1d1cd2594db\" (UID: \"07f53eed-c8ba-42ee-ac68-f1d1cd2594db\") " Sep 11 01:22:26.887452 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.887347 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-maas-api-tls\") pod \"07f53eed-c8ba-42ee-ac68-f1d1cd2594db\" (UID: \"07f53eed-c8ba-42ee-ac68-f1d1cd2594db\") " Sep 11 01:22:26.887576 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.887554 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "07f53eed-c8ba-42ee-ac68-f1d1cd2594db" (UID: "07f53eed-c8ba-42ee-ac68-f1d1cd2594db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:22:26.887707 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.887687 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-openshift-service-ca\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:26.889293 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.889262 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "07f53eed-c8ba-42ee-ac68-f1d1cd2594db" (UID: "07f53eed-c8ba-42ee-ac68-f1d1cd2594db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:22:26.889392 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.889289 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-kube-api-access-hxc6p" (OuterVolumeSpecName: "kube-api-access-hxc6p") pod "07f53eed-c8ba-42ee-ac68-f1d1cd2594db" (UID: "07f53eed-c8ba-42ee-ac68-f1d1cd2594db"). InnerVolumeSpecName "kube-api-access-hxc6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:22:26.889392 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.889332 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "07f53eed-c8ba-42ee-ac68-f1d1cd2594db" (UID: "07f53eed-c8ba-42ee-ac68-f1d1cd2594db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:22:26.989125 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.989100 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-metrics-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:26.989125 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.989121 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-maas-api-tls\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:26.989437 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:26.989130 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxc6p\" (UniqueName: \"kubernetes.io/projected/07f53eed-c8ba-42ee-ac68-f1d1cd2594db-kube-api-access-hxc6p\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:27.117416 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:27.117388 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk"] Sep 11 01:22:27.121419 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:27.121382 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-651569-79b5455c88-v4pzk"] Sep 11 01:22:27.136111 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:27.136087 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:27.678646 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:27.678611 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07f53eed-c8ba-42ee-ac68-f1d1cd2594db" path="/var/lib/kubelet/pods/07f53eed-c8ba-42ee-ac68-f1d1cd2594db/volumes" Sep 11 01:22:30.251812 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.251743 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" podUID="d1273e50-f71a-4b79-9ea5-654a64f60134" containerName="main" containerID="cri-o://41d0b77755cd942d87479922bb5ead3e2af75c5db12c49cd0fb904fe96bad6a9" gracePeriod=45 Sep 11 01:22:30.280480 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.280453 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" podUID="74c8f859-e529-4fe1-91f4-39f9a1be60fa" containerName="main" containerID="cri-o://2a958cebc5b05181555778dc00bc7b60f973bf1328f0de711d309b42d54a921b" gracePeriod=45 Sep 11 01:22:30.290787 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.290759 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" podUID="c47f9ceb-683c-40ff-8a63-8d0d8270b2d1" containerName="main" containerID="cri-o://5bc08d3af1555b2cf71ce36c16522f05439643bd44ca9123b83cb8061a086409" gracePeriod=45 Sep 11 01:22:30.591255 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.591219 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:22:30.594745 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.594720 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:22:30.599366 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.599346 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:22:30.721439 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.721406 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ql667\" (UniqueName: \"kubernetes.io/projected/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-kube-api-access-ql667\") pod \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\" (UID: \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\") " Sep 11 01:22:30.721627 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.721446 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74c8f859-e529-4fe1-91f4-39f9a1be60fa-model-cache\") pod \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\" (UID: \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\") " Sep 11 01:22:30.721627 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.721467 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74c8f859-e529-4fe1-91f4-39f9a1be60fa-home\") pod \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\" (UID: \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\") " Sep 11 01:22:30.721627 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.721485 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1273e50-f71a-4b79-9ea5-654a64f60134-tmp-dir\") pod \"d1273e50-f71a-4b79-9ea5-654a64f60134\" (UID: \"d1273e50-f71a-4b79-9ea5-654a64f60134\") " Sep 11 01:22:30.721627 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.721501 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-dshm\") pod \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\" (UID: \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\") " Sep 11 01:22:30.721627 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.721535 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74c8f859-e529-4fe1-91f4-39f9a1be60fa-dshm\") pod \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\" (UID: \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\") " Sep 11 01:22:30.721627 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.721566 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-tmp-dir\") pod \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\" (UID: \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\") " Sep 11 01:22:30.721627 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.721591 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74c8f859-e529-4fe1-91f4-39f9a1be60fa-tmp-dir\") pod \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\" (UID: \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\") " Sep 11 01:22:30.721627 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.721624 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74c8f859-e529-4fe1-91f4-39f9a1be60fa-tls-certs\") pod \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\" (UID: \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\") " Sep 11 01:22:30.722086 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.721652 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1273e50-f71a-4b79-9ea5-654a64f60134-dshm\") pod \"d1273e50-f71a-4b79-9ea5-654a64f60134\" (UID: \"d1273e50-f71a-4b79-9ea5-654a64f60134\") " Sep 11 01:22:30.722086 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.721690 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1273e50-f71a-4b79-9ea5-654a64f60134-tls-certs\") pod \"d1273e50-f71a-4b79-9ea5-654a64f60134\" (UID: \"d1273e50-f71a-4b79-9ea5-654a64f60134\") " Sep 11 01:22:30.722086 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.721693 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c8f859-e529-4fe1-91f4-39f9a1be60fa-home" (OuterVolumeSpecName: "home") pod "74c8f859-e529-4fe1-91f4-39f9a1be60fa" (UID: "74c8f859-e529-4fe1-91f4-39f9a1be60fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:22:30.722086 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.721706 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c8f859-e529-4fe1-91f4-39f9a1be60fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "74c8f859-e529-4fe1-91f4-39f9a1be60fa" (UID: "74c8f859-e529-4fe1-91f4-39f9a1be60fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:22:30.722086 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.721716 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1273e50-f71a-4b79-9ea5-654a64f60134-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d1273e50-f71a-4b79-9ea5-654a64f60134" (UID: "d1273e50-f71a-4b79-9ea5-654a64f60134"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:22:30.722086 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.721714 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-tls-certs\") pod \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\" (UID: \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\") " Sep 11 01:22:30.722086 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.721803 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xb44\" (UniqueName: \"kubernetes.io/projected/d1273e50-f71a-4b79-9ea5-654a64f60134-kube-api-access-5xb44\") pod \"d1273e50-f71a-4b79-9ea5-654a64f60134\" (UID: \"d1273e50-f71a-4b79-9ea5-654a64f60134\") " Sep 11 01:22:30.722086 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.721837 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1273e50-f71a-4b79-9ea5-654a64f60134-home\") pod \"d1273e50-f71a-4b79-9ea5-654a64f60134\" (UID: \"d1273e50-f71a-4b79-9ea5-654a64f60134\") " Sep 11 01:22:30.722086 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.721882 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-model-cache\") pod \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\" (UID: \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\") " Sep 11 01:22:30.722086 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.721927 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-home\") pod \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\" (UID: \"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1\") " Sep 11 01:22:30.722086 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.721970 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfk2v\" (UniqueName: \"kubernetes.io/projected/74c8f859-e529-4fe1-91f4-39f9a1be60fa-kube-api-access-nfk2v\") pod \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\" (UID: \"74c8f859-e529-4fe1-91f4-39f9a1be60fa\") " Sep 11 01:22:30.722086 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.722025 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c47f9ceb-683c-40ff-8a63-8d0d8270b2d1" (UID: "c47f9ceb-683c-40ff-8a63-8d0d8270b2d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:22:30.722086 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.722071 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1273e50-f71a-4b79-9ea5-654a64f60134-model-cache\") pod \"d1273e50-f71a-4b79-9ea5-654a64f60134\" (UID: \"d1273e50-f71a-4b79-9ea5-654a64f60134\") " Sep 11 01:22:30.722696 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.722307 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c8f859-e529-4fe1-91f4-39f9a1be60fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "74c8f859-e529-4fe1-91f4-39f9a1be60fa" (UID: "74c8f859-e529-4fe1-91f4-39f9a1be60fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:22:30.722696 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.722478 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1273e50-f71a-4b79-9ea5-654a64f60134-model-cache" (OuterVolumeSpecName: "model-cache") pod "d1273e50-f71a-4b79-9ea5-654a64f60134" (UID: "d1273e50-f71a-4b79-9ea5-654a64f60134"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:22:30.722696 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.722485 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74c8f859-e529-4fe1-91f4-39f9a1be60fa-model-cache\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:30.722696 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.722515 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74c8f859-e529-4fe1-91f4-39f9a1be60fa-home\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:30.722696 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.722530 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1273e50-f71a-4b79-9ea5-654a64f60134-tmp-dir\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:30.722696 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.722544 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-tmp-dir\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:30.722696 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.722556 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74c8f859-e529-4fe1-91f4-39f9a1be60fa-tmp-dir\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:30.722696 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.722659 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-model-cache" (OuterVolumeSpecName: "model-cache") pod "c47f9ceb-683c-40ff-8a63-8d0d8270b2d1" (UID: "c47f9ceb-683c-40ff-8a63-8d0d8270b2d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:22:30.722696 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.722696 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-home" (OuterVolumeSpecName: "home") pod "c47f9ceb-683c-40ff-8a63-8d0d8270b2d1" (UID: "c47f9ceb-683c-40ff-8a63-8d0d8270b2d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:22:30.724346 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.724312 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-dshm" (OuterVolumeSpecName: "dshm") pod "c47f9ceb-683c-40ff-8a63-8d0d8270b2d1" (UID: "c47f9ceb-683c-40ff-8a63-8d0d8270b2d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:22:30.724346 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.724325 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-kube-api-access-ql667" (OuterVolumeSpecName: "kube-api-access-ql667") pod "c47f9ceb-683c-40ff-8a63-8d0d8270b2d1" (UID: "c47f9ceb-683c-40ff-8a63-8d0d8270b2d1"). InnerVolumeSpecName "kube-api-access-ql667". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:22:30.724346 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.724333 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1273e50-f71a-4b79-9ea5-654a64f60134-dshm" (OuterVolumeSpecName: "dshm") pod "d1273e50-f71a-4b79-9ea5-654a64f60134" (UID: "d1273e50-f71a-4b79-9ea5-654a64f60134"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:22:30.724578 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.724393 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c8f859-e529-4fe1-91f4-39f9a1be60fa-dshm" (OuterVolumeSpecName: "dshm") pod "74c8f859-e529-4fe1-91f4-39f9a1be60fa" (UID: "74c8f859-e529-4fe1-91f4-39f9a1be60fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:22:30.724578 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.724515 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1273e50-f71a-4b79-9ea5-654a64f60134-home" (OuterVolumeSpecName: "home") pod "d1273e50-f71a-4b79-9ea5-654a64f60134" (UID: "d1273e50-f71a-4b79-9ea5-654a64f60134"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:22:30.724753 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.724728 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c47f9ceb-683c-40ff-8a63-8d0d8270b2d1" (UID: "c47f9ceb-683c-40ff-8a63-8d0d8270b2d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:22:30.724894 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.724874 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c8f859-e529-4fe1-91f4-39f9a1be60fa-kube-api-access-nfk2v" (OuterVolumeSpecName: "kube-api-access-nfk2v") pod "74c8f859-e529-4fe1-91f4-39f9a1be60fa" (UID: "74c8f859-e529-4fe1-91f4-39f9a1be60fa"). InnerVolumeSpecName "kube-api-access-nfk2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:22:30.725101 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.725075 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1273e50-f71a-4b79-9ea5-654a64f60134-kube-api-access-5xb44" (OuterVolumeSpecName: "kube-api-access-5xb44") pod "d1273e50-f71a-4b79-9ea5-654a64f60134" (UID: "d1273e50-f71a-4b79-9ea5-654a64f60134"). InnerVolumeSpecName "kube-api-access-5xb44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:22:30.725399 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.725385 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74c8f859-e529-4fe1-91f4-39f9a1be60fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "74c8f859-e529-4fe1-91f4-39f9a1be60fa" (UID: "74c8f859-e529-4fe1-91f4-39f9a1be60fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:22:30.726047 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.726032 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1273e50-f71a-4b79-9ea5-654a64f60134-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d1273e50-f71a-4b79-9ea5-654a64f60134" (UID: "d1273e50-f71a-4b79-9ea5-654a64f60134"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:22:30.819760 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.819696 2554 generic.go:356] "Generic (PLEG): container finished" podID="74c8f859-e529-4fe1-91f4-39f9a1be60fa" containerID="2a958cebc5b05181555778dc00bc7b60f973bf1328f0de711d309b42d54a921b" exitCode=0 Sep 11 01:22:30.819760 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.819734 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" Sep 11 01:22:30.819943 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.819775 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" event={"ID":"74c8f859-e529-4fe1-91f4-39f9a1be60fa","Type":"ContainerDied","Data":"2a958cebc5b05181555778dc00bc7b60f973bf1328f0de711d309b42d54a921b"} Sep 11 01:22:30.819943 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.819813 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv" event={"ID":"74c8f859-e529-4fe1-91f4-39f9a1be60fa","Type":"ContainerDied","Data":"c69c9837e0c93d8676df847130818bc098b2a0f3bd7dae8f12425b7cf9485fcb"} Sep 11 01:22:30.819943 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.819832 2554 scope.go:117] "RemoveContainer" containerID="2a958cebc5b05181555778dc00bc7b60f973bf1328f0de711d309b42d54a921b" Sep 11 01:22:30.821259 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.821242 2554 generic.go:356] "Generic (PLEG): container finished" podID="c47f9ceb-683c-40ff-8a63-8d0d8270b2d1" containerID="5bc08d3af1555b2cf71ce36c16522f05439643bd44ca9123b83cb8061a086409" exitCode=0 Sep 11 01:22:30.821366 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.821290 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" Sep 11 01:22:30.821366 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.821341 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" event={"ID":"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1","Type":"ContainerDied","Data":"5bc08d3af1555b2cf71ce36c16522f05439643bd44ca9123b83cb8061a086409"} Sep 11 01:22:30.821515 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.821386 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh" event={"ID":"c47f9ceb-683c-40ff-8a63-8d0d8270b2d1","Type":"ContainerDied","Data":"a4186bad173e5bad8ad35bbe380af899032627c4fddc13c786e1d1adcd3cd956"} Sep 11 01:22:30.822713 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.822690 2554 generic.go:356] "Generic (PLEG): container finished" podID="d1273e50-f71a-4b79-9ea5-654a64f60134" containerID="41d0b77755cd942d87479922bb5ead3e2af75c5db12c49cd0fb904fe96bad6a9" exitCode=0 Sep 11 01:22:30.822816 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.822726 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" Sep 11 01:22:30.822816 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.822772 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" event={"ID":"d1273e50-f71a-4b79-9ea5-654a64f60134","Type":"ContainerDied","Data":"41d0b77755cd942d87479922bb5ead3e2af75c5db12c49cd0fb904fe96bad6a9"} Sep 11 01:22:30.822816 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.822800 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r" event={"ID":"d1273e50-f71a-4b79-9ea5-654a64f60134","Type":"ContainerDied","Data":"d4c9a0481379e7e81bf44c47f13b4fab2e0026050e17b4ace9e7eb96aa042996"} Sep 11 01:22:30.823389 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.823198 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql667\" (UniqueName: \"kubernetes.io/projected/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-kube-api-access-ql667\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:30.823389 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.823223 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-dshm\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:30.823389 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.823236 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74c8f859-e529-4fe1-91f4-39f9a1be60fa-dshm\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:30.823389 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.823246 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74c8f859-e529-4fe1-91f4-39f9a1be60fa-tls-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:30.823389 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.823259 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1273e50-f71a-4b79-9ea5-654a64f60134-dshm\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:30.823389 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.823270 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1273e50-f71a-4b79-9ea5-654a64f60134-tls-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:30.823389 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.823280 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-tls-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:30.823389 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.823292 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xb44\" (UniqueName: \"kubernetes.io/projected/d1273e50-f71a-4b79-9ea5-654a64f60134-kube-api-access-5xb44\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:30.823389 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.823305 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1273e50-f71a-4b79-9ea5-654a64f60134-home\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:30.823389 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.823325 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-model-cache\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:30.823389 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.823337 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1-home\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:30.823389 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.823350 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfk2v\" (UniqueName: \"kubernetes.io/projected/74c8f859-e529-4fe1-91f4-39f9a1be60fa-kube-api-access-nfk2v\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:30.823389 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.823363 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1273e50-f71a-4b79-9ea5-654a64f60134-model-cache\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:30.830095 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.830075 2554 scope.go:117] "RemoveContainer" containerID="2a958cebc5b05181555778dc00bc7b60f973bf1328f0de711d309b42d54a921b" Sep 11 01:22:30.830391 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:22:30.830365 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a958cebc5b05181555778dc00bc7b60f973bf1328f0de711d309b42d54a921b\": container with ID starting with 2a958cebc5b05181555778dc00bc7b60f973bf1328f0de711d309b42d54a921b not found: ID does not exist" containerID="2a958cebc5b05181555778dc00bc7b60f973bf1328f0de711d309b42d54a921b" Sep 11 01:22:30.830462 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.830402 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a958cebc5b05181555778dc00bc7b60f973bf1328f0de711d309b42d54a921b"} err="failed to get container status \"2a958cebc5b05181555778dc00bc7b60f973bf1328f0de711d309b42d54a921b\": rpc error: code = NotFound desc = could not find container \"2a958cebc5b05181555778dc00bc7b60f973bf1328f0de711d309b42d54a921b\": container with ID starting with 2a958cebc5b05181555778dc00bc7b60f973bf1328f0de711d309b42d54a921b not found: ID does not exist" Sep 11 01:22:30.830462 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.830423 2554 scope.go:117] "RemoveContainer" containerID="5bc08d3af1555b2cf71ce36c16522f05439643bd44ca9123b83cb8061a086409" Sep 11 01:22:30.838789 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.838756 2554 scope.go:117] "RemoveContainer" containerID="5bc08d3af1555b2cf71ce36c16522f05439643bd44ca9123b83cb8061a086409" Sep 11 01:22:30.839039 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:22:30.839022 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc08d3af1555b2cf71ce36c16522f05439643bd44ca9123b83cb8061a086409\": container with ID starting with 5bc08d3af1555b2cf71ce36c16522f05439643bd44ca9123b83cb8061a086409 not found: ID does not exist" containerID="5bc08d3af1555b2cf71ce36c16522f05439643bd44ca9123b83cb8061a086409" Sep 11 01:22:30.839084 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.839046 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc08d3af1555b2cf71ce36c16522f05439643bd44ca9123b83cb8061a086409"} err="failed to get container status \"5bc08d3af1555b2cf71ce36c16522f05439643bd44ca9123b83cb8061a086409\": rpc error: code = NotFound desc = could not find container \"5bc08d3af1555b2cf71ce36c16522f05439643bd44ca9123b83cb8061a086409\": container with ID starting with 5bc08d3af1555b2cf71ce36c16522f05439643bd44ca9123b83cb8061a086409 not found: ID does not exist" Sep 11 01:22:30.839084 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.839061 2554 scope.go:117] "RemoveContainer" containerID="41d0b77755cd942d87479922bb5ead3e2af75c5db12c49cd0fb904fe96bad6a9" Sep 11 01:22:30.847427 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.847358 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv"] Sep 11 01:22:30.847483 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.847465 2554 scope.go:117] "RemoveContainer" containerID="41d0b77755cd942d87479922bb5ead3e2af75c5db12c49cd0fb904fe96bad6a9" Sep 11 01:22:30.847737 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:22:30.847715 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d0b77755cd942d87479922bb5ead3e2af75c5db12c49cd0fb904fe96bad6a9\": container with ID starting with 41d0b77755cd942d87479922bb5ead3e2af75c5db12c49cd0fb904fe96bad6a9 not found: ID does not exist" containerID="41d0b77755cd942d87479922bb5ead3e2af75c5db12c49cd0fb904fe96bad6a9" Sep 11 01:22:30.847776 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.847747 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d0b77755cd942d87479922bb5ead3e2af75c5db12c49cd0fb904fe96bad6a9"} err="failed to get container status \"41d0b77755cd942d87479922bb5ead3e2af75c5db12c49cd0fb904fe96bad6a9\": rpc error: code = NotFound desc = could not find container \"41d0b77755cd942d87479922bb5ead3e2af75c5db12c49cd0fb904fe96bad6a9\": container with ID starting with 41d0b77755cd942d87479922bb5ead3e2af75c5db12c49cd0fb904fe96bad6a9 not found: ID does not exist" Sep 11 01:22:30.851662 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.851640 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-651569/simulator-w4-651569-kserve-744844544c-sb5xv"] Sep 11 01:22:30.862087 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.862068 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh"] Sep 11 01:22:30.868032 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.868012 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-651569/unconfigured-w4-651569-kserve-848d7fd45d-9vmzh"] Sep 11 01:22:30.877532 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.877513 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r"] Sep 11 01:22:30.883277 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:30.883258 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-651569/distinct-w4-651569-kserve-bb5b69ddf-4pg8r"] Sep 11 01:22:31.679357 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:31.679325 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74c8f859-e529-4fe1-91f4-39f9a1be60fa" path="/var/lib/kubelet/pods/74c8f859-e529-4fe1-91f4-39f9a1be60fa/volumes" Sep 11 01:22:31.679975 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:31.679707 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c47f9ceb-683c-40ff-8a63-8d0d8270b2d1" path="/var/lib/kubelet/pods/c47f9ceb-683c-40ff-8a63-8d0d8270b2d1/volumes" Sep 11 01:22:31.680080 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:31.680066 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1273e50-f71a-4b79-9ea5-654a64f60134" path="/var/lib/kubelet/pods/d1273e50-f71a-4b79-9ea5-654a64f60134/volumes" Sep 11 01:22:32.730983 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:32.730952 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:33.197086 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:33.197057 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:33.298043 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:33.298011 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:33.474542 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:33.474458 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:35.817473 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:35.817432 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:41.441409 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.441364 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9"] Sep 11 01:22:41.441898 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.441609 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9" podUID="b36b03a6-107a-461b-8a93-013521b46c1f" containerName="maas-api" containerID="cri-o://668129cd484b5ce34bd9bf78f438ab74e2297ec356b0fe45634463faa2c0e18c" gracePeriod=30 Sep 11 01:22:41.459051 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.458990 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:41.524594 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.524560 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:41.585764 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.585713 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:41.650921 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.650888 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:41.686023 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.685999 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9" Sep 11 01:22:41.714034 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.713974 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:41.818368 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.818335 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b36b03a6-107a-461b-8a93-013521b46c1f-metrics-certs\") pod \"b36b03a6-107a-461b-8a93-013521b46c1f\" (UID: \"b36b03a6-107a-461b-8a93-013521b46c1f\") " Sep 11 01:22:41.818368 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.818369 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b36b03a6-107a-461b-8a93-013521b46c1f-maas-api-tls\") pod \"b36b03a6-107a-461b-8a93-013521b46c1f\" (UID: \"b36b03a6-107a-461b-8a93-013521b46c1f\") " Sep 11 01:22:41.818563 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.818480 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36b03a6-107a-461b-8a93-013521b46c1f-openshift-service-ca\") pod \"b36b03a6-107a-461b-8a93-013521b46c1f\" (UID: \"b36b03a6-107a-461b-8a93-013521b46c1f\") " Sep 11 01:22:41.818563 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.818551 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c76p2\" (UniqueName: \"kubernetes.io/projected/b36b03a6-107a-461b-8a93-013521b46c1f-kube-api-access-c76p2\") pod \"b36b03a6-107a-461b-8a93-013521b46c1f\" (UID: \"b36b03a6-107a-461b-8a93-013521b46c1f\") " Sep 11 01:22:41.818989 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.818957 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b36b03a6-107a-461b-8a93-013521b46c1f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b36b03a6-107a-461b-8a93-013521b46c1f" (UID: "b36b03a6-107a-461b-8a93-013521b46c1f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:22:41.820971 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.820940 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b36b03a6-107a-461b-8a93-013521b46c1f-kube-api-access-c76p2" (OuterVolumeSpecName: "kube-api-access-c76p2") pod "b36b03a6-107a-461b-8a93-013521b46c1f" (UID: "b36b03a6-107a-461b-8a93-013521b46c1f"). InnerVolumeSpecName "kube-api-access-c76p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:22:41.821095 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.821003 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b36b03a6-107a-461b-8a93-013521b46c1f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b36b03a6-107a-461b-8a93-013521b46c1f" (UID: "b36b03a6-107a-461b-8a93-013521b46c1f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:22:41.821168 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.821087 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b36b03a6-107a-461b-8a93-013521b46c1f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b36b03a6-107a-461b-8a93-013521b46c1f" (UID: "b36b03a6-107a-461b-8a93-013521b46c1f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:22:41.883745 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.883712 2554 generic.go:356] "Generic (PLEG): container finished" podID="b36b03a6-107a-461b-8a93-013521b46c1f" containerID="668129cd484b5ce34bd9bf78f438ab74e2297ec356b0fe45634463faa2c0e18c" exitCode=0 Sep 11 01:22:41.883902 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.883765 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9" Sep 11 01:22:41.883902 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.883797 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9" event={"ID":"b36b03a6-107a-461b-8a93-013521b46c1f","Type":"ContainerDied","Data":"668129cd484b5ce34bd9bf78f438ab74e2297ec356b0fe45634463faa2c0e18c"} Sep 11 01:22:41.883902 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.883867 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9" event={"ID":"b36b03a6-107a-461b-8a93-013521b46c1f","Type":"ContainerDied","Data":"ed597d997fde5c6b1ba774de8f98f08117fc5c1c8e2c3ec4232889034ffeb5ea"} Sep 11 01:22:41.883902 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.883894 2554 scope.go:117] "RemoveContainer" containerID="668129cd484b5ce34bd9bf78f438ab74e2297ec356b0fe45634463faa2c0e18c" Sep 11 01:22:41.893254 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.893238 2554 scope.go:117] "RemoveContainer" containerID="668129cd484b5ce34bd9bf78f438ab74e2297ec356b0fe45634463faa2c0e18c" Sep 11 01:22:41.893521 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:22:41.893501 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"668129cd484b5ce34bd9bf78f438ab74e2297ec356b0fe45634463faa2c0e18c\": container with ID starting with 668129cd484b5ce34bd9bf78f438ab74e2297ec356b0fe45634463faa2c0e18c not found: ID does not exist" containerID="668129cd484b5ce34bd9bf78f438ab74e2297ec356b0fe45634463faa2c0e18c" Sep 11 01:22:41.893565 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.893528 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"668129cd484b5ce34bd9bf78f438ab74e2297ec356b0fe45634463faa2c0e18c"} err="failed to get container status \"668129cd484b5ce34bd9bf78f438ab74e2297ec356b0fe45634463faa2c0e18c\": rpc error: code = NotFound desc = could not find container \"668129cd484b5ce34bd9bf78f438ab74e2297ec356b0fe45634463faa2c0e18c\": container with ID starting with 668129cd484b5ce34bd9bf78f438ab74e2297ec356b0fe45634463faa2c0e18c not found: ID does not exist" Sep 11 01:22:41.908329 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.908309 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9"] Sep 11 01:22:41.914812 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.914793 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f1f91d-56c7d6646f-68bp9"] Sep 11 01:22:41.919878 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.919829 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b36b03a6-107a-461b-8a93-013521b46c1f-openshift-service-ca\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:41.919878 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.919875 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c76p2\" (UniqueName: \"kubernetes.io/projected/b36b03a6-107a-461b-8a93-013521b46c1f-kube-api-access-c76p2\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:41.920011 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.919886 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b36b03a6-107a-461b-8a93-013521b46c1f-metrics-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:41.920011 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:41.919894 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b36b03a6-107a-461b-8a93-013521b46c1f-maas-api-tls\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:22:42.007630 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:42.007557 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:43.678377 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:43.678344 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b36b03a6-107a-461b-8a93-013521b46c1f" path="/var/lib/kubelet/pods/b36b03a6-107a-461b-8a93-013521b46c1f/volumes" Sep 11 01:22:47.717036 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:47.717001 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:47.778696 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:47.778661 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:47.970204 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:47.970119 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:48.538638 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:48.538598 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:51.648352 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:51.648306 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:51.749508 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:51.749473 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:53.697470 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:53.697444 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wtd8v_c2b320dd-43e9-435d-8e4f-2bca1f1233ff/ovn-acl-logging/0.log" Sep 11 01:22:53.699357 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:53.699336 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wtd8v_c2b320dd-43e9-435d-8e4f-2bca1f1233ff/ovn-acl-logging/0.log" Sep 11 01:22:53.969319 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:53.969251 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:22:58.473291 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:22:58.473255 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:03.850313 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:03.850275 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj"] Sep 11 01:23:03.852787 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:03.850529 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" podUID="ccc8fbc2-92a6-44b3-84b0-f064114cc151" containerName="maas-api" containerID="cri-o://fad06d0c65b450b2c72eb50113bb8d7baacf01127b9ba20115fa27b0d423b4cc" gracePeriod=30 Sep 11 01:23:03.879788 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:03.879756 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:03.949789 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:03.949704 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:04.002478 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:04.002443 2554 generic.go:356] "Generic (PLEG): container finished" podID="ccc8fbc2-92a6-44b3-84b0-f064114cc151" containerID="fad06d0c65b450b2c72eb50113bb8d7baacf01127b9ba20115fa27b0d423b4cc" exitCode=0 Sep 11 01:23:04.002637 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:04.002488 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" event={"ID":"ccc8fbc2-92a6-44b3-84b0-f064114cc151","Type":"ContainerDied","Data":"fad06d0c65b450b2c72eb50113bb8d7baacf01127b9ba20115fa27b0d423b4cc"} Sep 11 01:23:04.011937 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:04.011899 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:04.069955 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:04.069921 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:04.096208 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:04.096188 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" Sep 11 01:23:04.199513 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:04.199485 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccc8fbc2-92a6-44b3-84b0-f064114cc151-metrics-certs\") pod \"ccc8fbc2-92a6-44b3-84b0-f064114cc151\" (UID: \"ccc8fbc2-92a6-44b3-84b0-f064114cc151\") " Sep 11 01:23:04.199652 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:04.199577 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npp4l\" (UniqueName: \"kubernetes.io/projected/ccc8fbc2-92a6-44b3-84b0-f064114cc151-kube-api-access-npp4l\") pod \"ccc8fbc2-92a6-44b3-84b0-f064114cc151\" (UID: \"ccc8fbc2-92a6-44b3-84b0-f064114cc151\") " Sep 11 01:23:04.199700 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:04.199674 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccc8fbc2-92a6-44b3-84b0-f064114cc151-maas-api-tls\") pod \"ccc8fbc2-92a6-44b3-84b0-f064114cc151\" (UID: \"ccc8fbc2-92a6-44b3-84b0-f064114cc151\") " Sep 11 01:23:04.199758 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:04.199712 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc8fbc2-92a6-44b3-84b0-f064114cc151-openshift-service-ca\") pod \"ccc8fbc2-92a6-44b3-84b0-f064114cc151\" (UID: \"ccc8fbc2-92a6-44b3-84b0-f064114cc151\") " Sep 11 01:23:04.200098 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:04.200070 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc8fbc2-92a6-44b3-84b0-f064114cc151-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ccc8fbc2-92a6-44b3-84b0-f064114cc151" (UID: "ccc8fbc2-92a6-44b3-84b0-f064114cc151"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:23:04.201463 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:04.201434 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccc8fbc2-92a6-44b3-84b0-f064114cc151-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ccc8fbc2-92a6-44b3-84b0-f064114cc151" (UID: "ccc8fbc2-92a6-44b3-84b0-f064114cc151"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:23:04.201609 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:04.201590 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccc8fbc2-92a6-44b3-84b0-f064114cc151-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ccc8fbc2-92a6-44b3-84b0-f064114cc151" (UID: "ccc8fbc2-92a6-44b3-84b0-f064114cc151"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:23:04.201656 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:04.201612 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc8fbc2-92a6-44b3-84b0-f064114cc151-kube-api-access-npp4l" (OuterVolumeSpecName: "kube-api-access-npp4l") pod "ccc8fbc2-92a6-44b3-84b0-f064114cc151" (UID: "ccc8fbc2-92a6-44b3-84b0-f064114cc151"). InnerVolumeSpecName "kube-api-access-npp4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:23:04.301209 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:04.301181 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccc8fbc2-92a6-44b3-84b0-f064114cc151-maas-api-tls\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:23:04.301209 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:04.301208 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc8fbc2-92a6-44b3-84b0-f064114cc151-openshift-service-ca\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:23:04.301363 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:04.301222 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccc8fbc2-92a6-44b3-84b0-f064114cc151-metrics-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:23:04.301363 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:04.301234 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npp4l\" (UniqueName: \"kubernetes.io/projected/ccc8fbc2-92a6-44b3-84b0-f064114cc151-kube-api-access-npp4l\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:23:04.469095 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:04.469021 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:05.008948 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:05.008921 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" event={"ID":"ccc8fbc2-92a6-44b3-84b0-f064114cc151","Type":"ContainerDied","Data":"6aa647ebce9a0901696e2e67f7c49cc09babb8cceee29076f37f34a89a10d2a0"} Sep 11 01:23:05.009407 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:05.008958 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj" Sep 11 01:23:05.009407 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:05.008979 2554 scope.go:117] "RemoveContainer" containerID="fad06d0c65b450b2c72eb50113bb8d7baacf01127b9ba20115fa27b0d423b4cc" Sep 11 01:23:05.032049 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:05.032025 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj"] Sep 11 01:23:05.035655 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:05.035637 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-816605-99bf745ff-lbmjj"] Sep 11 01:23:05.678593 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:05.678565 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccc8fbc2-92a6-44b3-84b0-f064114cc151" path="/var/lib/kubelet/pods/ccc8fbc2-92a6-44b3-84b0-f064114cc151/volumes" Sep 11 01:23:10.171455 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:10.171418 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:10.317696 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:10.317663 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:10.389829 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:10.389781 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:10.614647 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:10.614569 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:11.152271 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:11.152233 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:21.425104 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:21.425063 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78"] Sep 11 01:23:21.425573 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:21.425289 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" podUID="7d195969-3723-46af-996f-765a6ee5fcc1" containerName="maas-api" containerID="cri-o://7350403fb1fa07ac992ee26daa621586e3b7dc67897c89c707c1c40efba92b52" gracePeriod=30 Sep 11 01:23:21.464723 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:21.464674 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:21.529707 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:21.529670 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:21.596487 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:21.596073 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:21.665597 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:21.665571 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:21.683177 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:21.683130 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" Sep 11 01:23:21.744176 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:21.744147 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4f4tt\" (UniqueName: \"kubernetes.io/projected/7d195969-3723-46af-996f-765a6ee5fcc1-kube-api-access-4f4tt\") pod \"7d195969-3723-46af-996f-765a6ee5fcc1\" (UID: \"7d195969-3723-46af-996f-765a6ee5fcc1\") " Sep 11 01:23:21.744289 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:21.744236 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d195969-3723-46af-996f-765a6ee5fcc1-maas-api-tls\") pod \"7d195969-3723-46af-996f-765a6ee5fcc1\" (UID: \"7d195969-3723-46af-996f-765a6ee5fcc1\") " Sep 11 01:23:21.744289 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:21.744278 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d195969-3723-46af-996f-765a6ee5fcc1-openshift-service-ca\") pod \"7d195969-3723-46af-996f-765a6ee5fcc1\" (UID: \"7d195969-3723-46af-996f-765a6ee5fcc1\") " Sep 11 01:23:21.744374 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:21.744339 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d195969-3723-46af-996f-765a6ee5fcc1-metrics-certs\") pod \"7d195969-3723-46af-996f-765a6ee5fcc1\" (UID: \"7d195969-3723-46af-996f-765a6ee5fcc1\") " Sep 11 01:23:21.744817 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:21.744774 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d195969-3723-46af-996f-765a6ee5fcc1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7d195969-3723-46af-996f-765a6ee5fcc1" (UID: "7d195969-3723-46af-996f-765a6ee5fcc1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:23:21.746344 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:21.746315 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d195969-3723-46af-996f-765a6ee5fcc1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7d195969-3723-46af-996f-765a6ee5fcc1" (UID: "7d195969-3723-46af-996f-765a6ee5fcc1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:23:21.746463 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:21.746443 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d195969-3723-46af-996f-765a6ee5fcc1-kube-api-access-4f4tt" (OuterVolumeSpecName: "kube-api-access-4f4tt") pod "7d195969-3723-46af-996f-765a6ee5fcc1" (UID: "7d195969-3723-46af-996f-765a6ee5fcc1"). InnerVolumeSpecName "kube-api-access-4f4tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:23:21.746928 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:21.746905 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d195969-3723-46af-996f-765a6ee5fcc1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7d195969-3723-46af-996f-765a6ee5fcc1" (UID: "7d195969-3723-46af-996f-765a6ee5fcc1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:23:21.845729 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:21.845693 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d195969-3723-46af-996f-765a6ee5fcc1-metrics-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:23:21.845729 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:21.845728 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f4tt\" (UniqueName: \"kubernetes.io/projected/7d195969-3723-46af-996f-765a6ee5fcc1-kube-api-access-4f4tt\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:23:21.845729 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:21.845739 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d195969-3723-46af-996f-765a6ee5fcc1-maas-api-tls\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:23:21.845948 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:21.845747 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d195969-3723-46af-996f-765a6ee5fcc1-openshift-service-ca\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:23:22.023595 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:22.023517 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:22.094597 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:22.094561 2554 generic.go:356] "Generic (PLEG): container finished" podID="7d195969-3723-46af-996f-765a6ee5fcc1" containerID="7350403fb1fa07ac992ee26daa621586e3b7dc67897c89c707c1c40efba92b52" exitCode=0 Sep 11 01:23:22.094730 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:22.094609 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" Sep 11 01:23:22.094730 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:22.094644 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" event={"ID":"7d195969-3723-46af-996f-765a6ee5fcc1","Type":"ContainerDied","Data":"7350403fb1fa07ac992ee26daa621586e3b7dc67897c89c707c1c40efba92b52"} Sep 11 01:23:22.094730 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:22.094696 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" event={"ID":"7d195969-3723-46af-996f-765a6ee5fcc1","Type":"ContainerDied","Data":"c8975d875a97b12730e512a260e018ff5ba2c7c8a6205f5f024c7da14177b625"} Sep 11 01:23:22.094730 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:22.094720 2554 scope.go:117] "RemoveContainer" containerID="7350403fb1fa07ac992ee26daa621586e3b7dc67897c89c707c1c40efba92b52" Sep 11 01:23:22.103694 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:22.103679 2554 scope.go:117] "RemoveContainer" containerID="7350403fb1fa07ac992ee26daa621586e3b7dc67897c89c707c1c40efba92b52" Sep 11 01:23:22.103958 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:23:22.103939 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7350403fb1fa07ac992ee26daa621586e3b7dc67897c89c707c1c40efba92b52\": container with ID starting with 7350403fb1fa07ac992ee26daa621586e3b7dc67897c89c707c1c40efba92b52 not found: ID does not exist" containerID="7350403fb1fa07ac992ee26daa621586e3b7dc67897c89c707c1c40efba92b52" Sep 11 01:23:22.104014 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:22.103966 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7350403fb1fa07ac992ee26daa621586e3b7dc67897c89c707c1c40efba92b52"} err="failed to get container status \"7350403fb1fa07ac992ee26daa621586e3b7dc67897c89c707c1c40efba92b52\": rpc error: code = NotFound desc = could not find container \"7350403fb1fa07ac992ee26daa621586e3b7dc67897c89c707c1c40efba92b52\": container with ID starting with 7350403fb1fa07ac992ee26daa621586e3b7dc67897c89c707c1c40efba92b52 not found: ID does not exist" Sep 11 01:23:22.115765 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:22.115745 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78"] Sep 11 01:23:22.119218 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:22.119199 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78"] Sep 11 01:23:23.677793 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:23.677760 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d195969-3723-46af-996f-765a6ee5fcc1" path="/var/lib/kubelet/pods/7d195969-3723-46af-996f-765a6ee5fcc1/volumes" Sep 11 01:23:24.662869 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:24.662804 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-799002-758f74bc98-hqz78" podUID="7d195969-3723-46af-996f-765a6ee5fcc1" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.57:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Sep 11 01:23:27.834646 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:27.834612 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:28.869382 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:28.869338 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:38.895587 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:38.895551 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278"] Sep 11 01:23:38.896107 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:38.895760 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" podUID="145ea3e7-21d8-415e-b81c-e534bf889a76" containerName="maas-api" containerID="cri-o://341c1f8a21f94023db4c1e3e8ff62b094120a8037e0bd8ea4bebeb50a54ba020" gracePeriod=30 Sep 11 01:23:38.920254 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:38.920225 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:39.002006 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.001868 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:39.069973 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.069939 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:39.138199 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.138169 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:39.157777 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.157727 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" Sep 11 01:23:39.179024 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.178995 2554 generic.go:356] "Generic (PLEG): container finished" podID="145ea3e7-21d8-415e-b81c-e534bf889a76" containerID="341c1f8a21f94023db4c1e3e8ff62b094120a8037e0bd8ea4bebeb50a54ba020" exitCode=0 Sep 11 01:23:39.179222 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.179042 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" Sep 11 01:23:39.179407 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.179063 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" event={"ID":"145ea3e7-21d8-415e-b81c-e534bf889a76","Type":"ContainerDied","Data":"341c1f8a21f94023db4c1e3e8ff62b094120a8037e0bd8ea4bebeb50a54ba020"} Sep 11 01:23:39.179532 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.179479 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278" event={"ID":"145ea3e7-21d8-415e-b81c-e534bf889a76","Type":"ContainerDied","Data":"7cf85b9e16384e73bae89075ed8ac2a3a410fdf988c8f72955f042ad1b792817"} Sep 11 01:23:39.179532 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.179517 2554 scope.go:117] "RemoveContainer" containerID="341c1f8a21f94023db4c1e3e8ff62b094120a8037e0bd8ea4bebeb50a54ba020" Sep 11 01:23:39.189729 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.189710 2554 scope.go:117] "RemoveContainer" containerID="341c1f8a21f94023db4c1e3e8ff62b094120a8037e0bd8ea4bebeb50a54ba020" Sep 11 01:23:39.190021 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:23:39.190002 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"341c1f8a21f94023db4c1e3e8ff62b094120a8037e0bd8ea4bebeb50a54ba020\": container with ID starting with 341c1f8a21f94023db4c1e3e8ff62b094120a8037e0bd8ea4bebeb50a54ba020 not found: ID does not exist" containerID="341c1f8a21f94023db4c1e3e8ff62b094120a8037e0bd8ea4bebeb50a54ba020" Sep 11 01:23:39.190107 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.190033 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341c1f8a21f94023db4c1e3e8ff62b094120a8037e0bd8ea4bebeb50a54ba020"} err="failed to get container status \"341c1f8a21f94023db4c1e3e8ff62b094120a8037e0bd8ea4bebeb50a54ba020\": rpc error: code = NotFound desc = could not find container \"341c1f8a21f94023db4c1e3e8ff62b094120a8037e0bd8ea4bebeb50a54ba020\": container with ID starting with 341c1f8a21f94023db4c1e3e8ff62b094120a8037e0bd8ea4bebeb50a54ba020 not found: ID does not exist" Sep 11 01:23:39.290298 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.290273 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145ea3e7-21d8-415e-b81c-e534bf889a76-maas-api-tls\") pod \"145ea3e7-21d8-415e-b81c-e534bf889a76\" (UID: \"145ea3e7-21d8-415e-b81c-e534bf889a76\") " Sep 11 01:23:39.290424 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.290364 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrqkl\" (UniqueName: \"kubernetes.io/projected/145ea3e7-21d8-415e-b81c-e534bf889a76-kube-api-access-lrqkl\") pod \"145ea3e7-21d8-415e-b81c-e534bf889a76\" (UID: \"145ea3e7-21d8-415e-b81c-e534bf889a76\") " Sep 11 01:23:39.290489 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.290455 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145ea3e7-21d8-415e-b81c-e534bf889a76-metrics-certs\") pod \"145ea3e7-21d8-415e-b81c-e534bf889a76\" (UID: \"145ea3e7-21d8-415e-b81c-e534bf889a76\") " Sep 11 01:23:39.290548 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.290486 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145ea3e7-21d8-415e-b81c-e534bf889a76-openshift-service-ca\") pod \"145ea3e7-21d8-415e-b81c-e534bf889a76\" (UID: \"145ea3e7-21d8-415e-b81c-e534bf889a76\") " Sep 11 01:23:39.290952 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.290907 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/145ea3e7-21d8-415e-b81c-e534bf889a76-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "145ea3e7-21d8-415e-b81c-e534bf889a76" (UID: "145ea3e7-21d8-415e-b81c-e534bf889a76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:23:39.292565 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.292537 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145ea3e7-21d8-415e-b81c-e534bf889a76-kube-api-access-lrqkl" (OuterVolumeSpecName: "kube-api-access-lrqkl") pod "145ea3e7-21d8-415e-b81c-e534bf889a76" (UID: "145ea3e7-21d8-415e-b81c-e534bf889a76"). InnerVolumeSpecName "kube-api-access-lrqkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:23:39.292826 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.292805 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/145ea3e7-21d8-415e-b81c-e534bf889a76-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "145ea3e7-21d8-415e-b81c-e534bf889a76" (UID: "145ea3e7-21d8-415e-b81c-e534bf889a76"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:23:39.292947 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.292832 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/145ea3e7-21d8-415e-b81c-e534bf889a76-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "145ea3e7-21d8-415e-b81c-e534bf889a76" (UID: "145ea3e7-21d8-415e-b81c-e534bf889a76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:23:39.391927 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.391900 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/145ea3e7-21d8-415e-b81c-e534bf889a76-metrics-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:23:39.391927 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.391924 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/145ea3e7-21d8-415e-b81c-e534bf889a76-openshift-service-ca\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:23:39.392066 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.391935 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/145ea3e7-21d8-415e-b81c-e534bf889a76-maas-api-tls\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:23:39.392066 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.391944 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrqkl\" (UniqueName: \"kubernetes.io/projected/145ea3e7-21d8-415e-b81c-e534bf889a76-kube-api-access-lrqkl\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:23:39.499970 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.499937 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:39.506018 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.505997 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278"] Sep 11 01:23:39.511748 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.511725 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-71ba6d-56db84b49c-mp278"] Sep 11 01:23:39.677939 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:39.677908 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="145ea3e7-21d8-415e-b81c-e534bf889a76" path="/var/lib/kubelet/pods/145ea3e7-21d8-415e-b81c-e534bf889a76/volumes" Sep 11 01:23:45.429342 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:45.429296 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:45.515544 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:45.515514 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:23:45.696080 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:23:45.696002 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:24:25.448856 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:25.448812 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct"] Sep 11 01:24:25.473481 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:25.473444 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:24:26.386041 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:26.386008 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn"] Sep 11 01:24:26.406149 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:26.406120 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:24:40.467763 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:40.467721 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" podUID="a2c1c1cd-8a80-4738-ac30-604ee1744a2f" containerName="main" containerID="cri-o://df0b406e55d95a0054815018d1024ee31101f0bb5e3d82f31b4229250c522745" gracePeriod=45 Sep 11 01:24:40.717600 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:40.717576 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:24:40.883112 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:40.883045 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9clt\" (UniqueName: \"kubernetes.io/projected/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-kube-api-access-v9clt\") pod \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\" (UID: \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\") " Sep 11 01:24:40.883112 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:40.883080 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-dshm\") pod \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\" (UID: \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\") " Sep 11 01:24:40.883312 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:40.883134 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-tls-certs\") pod \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\" (UID: \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\") " Sep 11 01:24:40.883312 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:40.883182 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-home\") pod \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\" (UID: \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\") " Sep 11 01:24:40.883312 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:40.883296 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-tmp-dir\") pod \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\" (UID: \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\") " Sep 11 01:24:40.883471 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:40.883379 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-model-cache\") pod \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\" (UID: \"a2c1c1cd-8a80-4738-ac30-604ee1744a2f\") " Sep 11 01:24:40.883471 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:40.883420 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-home" (OuterVolumeSpecName: "home") pod "a2c1c1cd-8a80-4738-ac30-604ee1744a2f" (UID: "a2c1c1cd-8a80-4738-ac30-604ee1744a2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:24:40.883586 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:40.883562 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2c1c1cd-8a80-4738-ac30-604ee1744a2f" (UID: "a2c1c1cd-8a80-4738-ac30-604ee1744a2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:24:40.883639 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:40.883593 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2c1c1cd-8a80-4738-ac30-604ee1744a2f" (UID: "a2c1c1cd-8a80-4738-ac30-604ee1744a2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:24:40.883713 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:40.883699 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-home\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:24:40.883765 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:40.883716 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-tmp-dir\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:24:40.883765 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:40.883730 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-model-cache\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:24:40.885186 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:40.885165 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-kube-api-access-v9clt" (OuterVolumeSpecName: "kube-api-access-v9clt") pod "a2c1c1cd-8a80-4738-ac30-604ee1744a2f" (UID: "a2c1c1cd-8a80-4738-ac30-604ee1744a2f"). InnerVolumeSpecName "kube-api-access-v9clt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:24:40.885575 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:40.885550 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2c1c1cd-8a80-4738-ac30-604ee1744a2f" (UID: "a2c1c1cd-8a80-4738-ac30-604ee1744a2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:24:40.885575 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:40.885560 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-dshm" (OuterVolumeSpecName: "dshm") pod "a2c1c1cd-8a80-4738-ac30-604ee1744a2f" (UID: "a2c1c1cd-8a80-4738-ac30-604ee1744a2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:24:40.984595 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:40.984562 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-tls-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:24:40.984595 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:40.984589 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9clt\" (UniqueName: \"kubernetes.io/projected/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-kube-api-access-v9clt\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:24:40.984773 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:40.984602 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2c1c1cd-8a80-4738-ac30-604ee1744a2f-dshm\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:24:41.409005 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.408943 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" podUID="983daa40-2b81-4a89-9e72-a719b5d93eb6" containerName="main" containerID="cri-o://cc6704a62d0c36b0951498568b555908df2e04f12f7ddde943ddd9dfcd366d86" gracePeriod=45 Sep 11 01:24:41.484865 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.484822 2554 generic.go:356] "Generic (PLEG): container finished" podID="a2c1c1cd-8a80-4738-ac30-604ee1744a2f" containerID="df0b406e55d95a0054815018d1024ee31101f0bb5e3d82f31b4229250c522745" exitCode=0 Sep 11 01:24:41.485176 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.484887 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" Sep 11 01:24:41.485176 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.484899 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" event={"ID":"a2c1c1cd-8a80-4738-ac30-604ee1744a2f","Type":"ContainerDied","Data":"df0b406e55d95a0054815018d1024ee31101f0bb5e3d82f31b4229250c522745"} Sep 11 01:24:41.485176 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.484948 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct" event={"ID":"a2c1c1cd-8a80-4738-ac30-604ee1744a2f","Type":"ContainerDied","Data":"1733a932c47ecd2b5544d1c5f6b6fd32cbdd6bf035f7af0d8af60f472c7e9995"} Sep 11 01:24:41.485176 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.484967 2554 scope.go:117] "RemoveContainer" containerID="df0b406e55d95a0054815018d1024ee31101f0bb5e3d82f31b4229250c522745" Sep 11 01:24:41.505351 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.505330 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct"] Sep 11 01:24:41.509418 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.509398 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2482ac/auth-test-model-2482ac-kserve-84ccf6b46c-w5fct"] Sep 11 01:24:41.521633 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.521617 2554 scope.go:117] "RemoveContainer" containerID="df0b406e55d95a0054815018d1024ee31101f0bb5e3d82f31b4229250c522745" Sep 11 01:24:41.521957 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:24:41.521936 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df0b406e55d95a0054815018d1024ee31101f0bb5e3d82f31b4229250c522745\": container with ID starting with df0b406e55d95a0054815018d1024ee31101f0bb5e3d82f31b4229250c522745 not found: ID does not exist" containerID="df0b406e55d95a0054815018d1024ee31101f0bb5e3d82f31b4229250c522745" Sep 11 01:24:41.522048 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.521963 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0b406e55d95a0054815018d1024ee31101f0bb5e3d82f31b4229250c522745"} err="failed to get container status \"df0b406e55d95a0054815018d1024ee31101f0bb5e3d82f31b4229250c522745\": rpc error: code = NotFound desc = could not find container \"df0b406e55d95a0054815018d1024ee31101f0bb5e3d82f31b4229250c522745\": container with ID starting with df0b406e55d95a0054815018d1024ee31101f0bb5e3d82f31b4229250c522745 not found: ID does not exist" Sep 11 01:24:41.666647 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.666594 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:24:41.677980 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.677955 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2c1c1cd-8a80-4738-ac30-604ee1744a2f" path="/var/lib/kubelet/pods/a2c1c1cd-8a80-4738-ac30-604ee1744a2f/volumes" Sep 11 01:24:41.789834 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.789807 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/983daa40-2b81-4a89-9e72-a719b5d93eb6-model-cache\") pod \"983daa40-2b81-4a89-9e72-a719b5d93eb6\" (UID: \"983daa40-2b81-4a89-9e72-a719b5d93eb6\") " Sep 11 01:24:41.790004 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.789874 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983daa40-2b81-4a89-9e72-a719b5d93eb6-tmp-dir\") pod \"983daa40-2b81-4a89-9e72-a719b5d93eb6\" (UID: \"983daa40-2b81-4a89-9e72-a719b5d93eb6\") " Sep 11 01:24:41.790004 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.789915 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/983daa40-2b81-4a89-9e72-a719b5d93eb6-home\") pod \"983daa40-2b81-4a89-9e72-a719b5d93eb6\" (UID: \"983daa40-2b81-4a89-9e72-a719b5d93eb6\") " Sep 11 01:24:41.790004 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.789954 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/983daa40-2b81-4a89-9e72-a719b5d93eb6-tls-certs\") pod \"983daa40-2b81-4a89-9e72-a719b5d93eb6\" (UID: \"983daa40-2b81-4a89-9e72-a719b5d93eb6\") " Sep 11 01:24:41.790175 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.790045 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klkjn\" (UniqueName: \"kubernetes.io/projected/983daa40-2b81-4a89-9e72-a719b5d93eb6-kube-api-access-klkjn\") pod \"983daa40-2b81-4a89-9e72-a719b5d93eb6\" (UID: \"983daa40-2b81-4a89-9e72-a719b5d93eb6\") " Sep 11 01:24:41.790175 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.790055 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983daa40-2b81-4a89-9e72-a719b5d93eb6-model-cache" (OuterVolumeSpecName: "model-cache") pod "983daa40-2b81-4a89-9e72-a719b5d93eb6" (UID: "983daa40-2b81-4a89-9e72-a719b5d93eb6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:24:41.790175 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.790091 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/983daa40-2b81-4a89-9e72-a719b5d93eb6-dshm\") pod \"983daa40-2b81-4a89-9e72-a719b5d93eb6\" (UID: \"983daa40-2b81-4a89-9e72-a719b5d93eb6\") " Sep 11 01:24:41.790327 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.790183 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983daa40-2b81-4a89-9e72-a719b5d93eb6-home" (OuterVolumeSpecName: "home") pod "983daa40-2b81-4a89-9e72-a719b5d93eb6" (UID: "983daa40-2b81-4a89-9e72-a719b5d93eb6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:24:41.790327 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.790210 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983daa40-2b81-4a89-9e72-a719b5d93eb6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "983daa40-2b81-4a89-9e72-a719b5d93eb6" (UID: "983daa40-2b81-4a89-9e72-a719b5d93eb6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:24:41.790446 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.790428 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983daa40-2b81-4a89-9e72-a719b5d93eb6-tmp-dir\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:24:41.790497 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.790451 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/983daa40-2b81-4a89-9e72-a719b5d93eb6-home\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:24:41.790497 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.790465 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/983daa40-2b81-4a89-9e72-a719b5d93eb6-model-cache\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:24:41.792219 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.792197 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/983daa40-2b81-4a89-9e72-a719b5d93eb6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "983daa40-2b81-4a89-9e72-a719b5d93eb6" (UID: "983daa40-2b81-4a89-9e72-a719b5d93eb6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:24:41.792317 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.792268 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/983daa40-2b81-4a89-9e72-a719b5d93eb6-dshm" (OuterVolumeSpecName: "dshm") pod "983daa40-2b81-4a89-9e72-a719b5d93eb6" (UID: "983daa40-2b81-4a89-9e72-a719b5d93eb6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:24:41.792317 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.792274 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983daa40-2b81-4a89-9e72-a719b5d93eb6-kube-api-access-klkjn" (OuterVolumeSpecName: "kube-api-access-klkjn") pod "983daa40-2b81-4a89-9e72-a719b5d93eb6" (UID: "983daa40-2b81-4a89-9e72-a719b5d93eb6"). InnerVolumeSpecName "kube-api-access-klkjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:24:41.891831 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.891809 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/983daa40-2b81-4a89-9e72-a719b5d93eb6-tls-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:24:41.891831 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.891829 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klkjn\" (UniqueName: \"kubernetes.io/projected/983daa40-2b81-4a89-9e72-a719b5d93eb6-kube-api-access-klkjn\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:24:41.891983 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:41.891852 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/983daa40-2b81-4a89-9e72-a719b5d93eb6-dshm\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:24:42.497201 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:42.497158 2554 generic.go:356] "Generic (PLEG): container finished" podID="983daa40-2b81-4a89-9e72-a719b5d93eb6" containerID="cc6704a62d0c36b0951498568b555908df2e04f12f7ddde943ddd9dfcd366d86" exitCode=0 Sep 11 01:24:42.497668 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:42.497225 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" Sep 11 01:24:42.497668 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:42.497232 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" event={"ID":"983daa40-2b81-4a89-9e72-a719b5d93eb6","Type":"ContainerDied","Data":"cc6704a62d0c36b0951498568b555908df2e04f12f7ddde943ddd9dfcd366d86"} Sep 11 01:24:42.497668 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:42.497268 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn" event={"ID":"983daa40-2b81-4a89-9e72-a719b5d93eb6","Type":"ContainerDied","Data":"1a539893d63e41b75e37bd404ba71cd96375e36ab4c91aea694e880094fbfb30"} Sep 11 01:24:42.497668 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:42.497285 2554 scope.go:117] "RemoveContainer" containerID="cc6704a62d0c36b0951498568b555908df2e04f12f7ddde943ddd9dfcd366d86" Sep 11 01:24:42.506442 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:42.506423 2554 scope.go:117] "RemoveContainer" containerID="cc6704a62d0c36b0951498568b555908df2e04f12f7ddde943ddd9dfcd366d86" Sep 11 01:24:42.506663 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:24:42.506645 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc6704a62d0c36b0951498568b555908df2e04f12f7ddde943ddd9dfcd366d86\": container with ID starting with cc6704a62d0c36b0951498568b555908df2e04f12f7ddde943ddd9dfcd366d86 not found: ID does not exist" containerID="cc6704a62d0c36b0951498568b555908df2e04f12f7ddde943ddd9dfcd366d86" Sep 11 01:24:42.506717 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:42.506671 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc6704a62d0c36b0951498568b555908df2e04f12f7ddde943ddd9dfcd366d86"} err="failed to get container status \"cc6704a62d0c36b0951498568b555908df2e04f12f7ddde943ddd9dfcd366d86\": rpc error: code = NotFound desc = could not find container \"cc6704a62d0c36b0951498568b555908df2e04f12f7ddde943ddd9dfcd366d86\": container with ID starting with cc6704a62d0c36b0951498568b555908df2e04f12f7ddde943ddd9dfcd366d86 not found: ID does not exist" Sep 11 01:24:42.518656 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:42.518627 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn"] Sep 11 01:24:42.522342 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:42.522323 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9433b9/auth-test-model-9433b9-kserve-7fbdbc9ddc-g5knn"] Sep 11 01:24:43.105374 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105346 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc"] Sep 11 01:24:43.105709 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105697 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74c8f859-e529-4fe1-91f4-39f9a1be60fa" containerName="main" Sep 11 01:24:43.105758 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105711 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="74c8f859-e529-4fe1-91f4-39f9a1be60fa" containerName="main" Sep 11 01:24:43.105758 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105723 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="145ea3e7-21d8-415e-b81c-e534bf889a76" containerName="maas-api" Sep 11 01:24:43.105758 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105729 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="145ea3e7-21d8-415e-b81c-e534bf889a76" containerName="maas-api" Sep 11 01:24:43.105758 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105739 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b36b03a6-107a-461b-8a93-013521b46c1f" containerName="maas-api" Sep 11 01:24:43.105758 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105744 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b36b03a6-107a-461b-8a93-013521b46c1f" containerName="maas-api" Sep 11 01:24:43.105758 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105754 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c47f9ceb-683c-40ff-8a63-8d0d8270b2d1" containerName="main" Sep 11 01:24:43.105758 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105759 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c47f9ceb-683c-40ff-8a63-8d0d8270b2d1" containerName="main" Sep 11 01:24:43.105989 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105766 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1273e50-f71a-4b79-9ea5-654a64f60134" containerName="main" Sep 11 01:24:43.105989 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105771 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1273e50-f71a-4b79-9ea5-654a64f60134" containerName="main" Sep 11 01:24:43.105989 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105780 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2c1c1cd-8a80-4738-ac30-604ee1744a2f" containerName="main" Sep 11 01:24:43.105989 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105785 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2c1c1cd-8a80-4738-ac30-604ee1744a2f" containerName="main" Sep 11 01:24:43.105989 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105792 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d195969-3723-46af-996f-765a6ee5fcc1" containerName="maas-api" Sep 11 01:24:43.105989 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105797 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d195969-3723-46af-996f-765a6ee5fcc1" containerName="maas-api" Sep 11 01:24:43.105989 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105805 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07f53eed-c8ba-42ee-ac68-f1d1cd2594db" containerName="maas-api" Sep 11 01:24:43.105989 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105809 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="07f53eed-c8ba-42ee-ac68-f1d1cd2594db" containerName="maas-api" Sep 11 01:24:43.105989 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105816 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccc8fbc2-92a6-44b3-84b0-f064114cc151" containerName="maas-api" Sep 11 01:24:43.105989 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105820 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccc8fbc2-92a6-44b3-84b0-f064114cc151" containerName="maas-api" Sep 11 01:24:43.105989 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105826 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="983daa40-2b81-4a89-9e72-a719b5d93eb6" containerName="main" Sep 11 01:24:43.105989 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105831 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="983daa40-2b81-4a89-9e72-a719b5d93eb6" containerName="main" Sep 11 01:24:43.105989 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105904 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="145ea3e7-21d8-415e-b81c-e534bf889a76" containerName="maas-api" Sep 11 01:24:43.105989 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105913 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="07f53eed-c8ba-42ee-ac68-f1d1cd2594db" containerName="maas-api" Sep 11 01:24:43.105989 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105920 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2c1c1cd-8a80-4738-ac30-604ee1744a2f" containerName="main" Sep 11 01:24:43.105989 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105926 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b36b03a6-107a-461b-8a93-013521b46c1f" containerName="maas-api" Sep 11 01:24:43.105989 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105933 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccc8fbc2-92a6-44b3-84b0-f064114cc151" containerName="maas-api" Sep 11 01:24:43.105989 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105938 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1273e50-f71a-4b79-9ea5-654a64f60134" containerName="main" Sep 11 01:24:43.105989 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105943 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="74c8f859-e529-4fe1-91f4-39f9a1be60fa" containerName="main" Sep 11 01:24:43.105989 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105949 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d195969-3723-46af-996f-765a6ee5fcc1" containerName="maas-api" Sep 11 01:24:43.105989 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105954 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c47f9ceb-683c-40ff-8a63-8d0d8270b2d1" containerName="main" Sep 11 01:24:43.105989 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.105959 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="983daa40-2b81-4a89-9e72-a719b5d93eb6" containerName="main" Sep 11 01:24:43.110265 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.110246 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:24:43.113469 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.113446 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2482ac\"/\"openshift-service-ca.crt\"" Sep 11 01:24:43.113597 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.113549 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2482ac\"/\"sub-test-model-2482ac-kserve-self-signed-certs\"" Sep 11 01:24:43.114361 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.113446 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2482ac\"/\"default-dockercfg-w9chz\"" Sep 11 01:24:43.114361 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.113783 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2482ac\"/\"kube-root-ca.crt\"" Sep 11 01:24:43.115781 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.115758 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc"] Sep 11 01:24:43.124885 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.124836 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:24:43.201107 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.201082 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/477c529e-b2ce-4dca-ab1c-987efbade140-home\") pod \"sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc\" (UID: \"477c529e-b2ce-4dca-ab1c-987efbade140\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:24:43.201217 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.201126 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/477c529e-b2ce-4dca-ab1c-987efbade140-dshm\") pod \"sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc\" (UID: \"477c529e-b2ce-4dca-ab1c-987efbade140\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:24:43.201217 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.201157 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/477c529e-b2ce-4dca-ab1c-987efbade140-model-cache\") pod \"sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc\" (UID: \"477c529e-b2ce-4dca-ab1c-987efbade140\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:24:43.201293 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.201232 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/477c529e-b2ce-4dca-ab1c-987efbade140-tmp-dir\") pod \"sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc\" (UID: \"477c529e-b2ce-4dca-ab1c-987efbade140\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:24:43.201336 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.201309 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/477c529e-b2ce-4dca-ab1c-987efbade140-tls-certs\") pod \"sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc\" (UID: \"477c529e-b2ce-4dca-ab1c-987efbade140\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:24:43.201375 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.201340 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7w27\" (UniqueName: \"kubernetes.io/projected/477c529e-b2ce-4dca-ab1c-987efbade140-kube-api-access-f7w27\") pod \"sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc\" (UID: \"477c529e-b2ce-4dca-ab1c-987efbade140\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:24:43.302547 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.302521 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/477c529e-b2ce-4dca-ab1c-987efbade140-tls-certs\") pod \"sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc\" (UID: \"477c529e-b2ce-4dca-ab1c-987efbade140\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:24:43.302644 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.302555 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7w27\" (UniqueName: \"kubernetes.io/projected/477c529e-b2ce-4dca-ab1c-987efbade140-kube-api-access-f7w27\") pod \"sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc\" (UID: \"477c529e-b2ce-4dca-ab1c-987efbade140\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:24:43.302644 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.302580 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/477c529e-b2ce-4dca-ab1c-987efbade140-home\") pod \"sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc\" (UID: \"477c529e-b2ce-4dca-ab1c-987efbade140\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:24:43.302644 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.302604 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/477c529e-b2ce-4dca-ab1c-987efbade140-dshm\") pod \"sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc\" (UID: \"477c529e-b2ce-4dca-ab1c-987efbade140\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:24:43.302644 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.302619 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/477c529e-b2ce-4dca-ab1c-987efbade140-model-cache\") pod \"sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc\" (UID: \"477c529e-b2ce-4dca-ab1c-987efbade140\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:24:43.302819 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.302646 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/477c529e-b2ce-4dca-ab1c-987efbade140-tmp-dir\") pod \"sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc\" (UID: \"477c529e-b2ce-4dca-ab1c-987efbade140\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:24:43.303035 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.303007 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/477c529e-b2ce-4dca-ab1c-987efbade140-home\") pod \"sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc\" (UID: \"477c529e-b2ce-4dca-ab1c-987efbade140\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:24:43.303116 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.303077 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/477c529e-b2ce-4dca-ab1c-987efbade140-tmp-dir\") pod \"sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc\" (UID: \"477c529e-b2ce-4dca-ab1c-987efbade140\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:24:43.303116 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.303098 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/477c529e-b2ce-4dca-ab1c-987efbade140-model-cache\") pod \"sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc\" (UID: \"477c529e-b2ce-4dca-ab1c-987efbade140\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:24:43.304887 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.304871 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/477c529e-b2ce-4dca-ab1c-987efbade140-dshm\") pod \"sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc\" (UID: \"477c529e-b2ce-4dca-ab1c-987efbade140\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:24:43.305021 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.305004 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/477c529e-b2ce-4dca-ab1c-987efbade140-tls-certs\") pod \"sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc\" (UID: \"477c529e-b2ce-4dca-ab1c-987efbade140\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:24:43.310341 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.310320 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7w27\" (UniqueName: \"kubernetes.io/projected/477c529e-b2ce-4dca-ab1c-987efbade140-kube-api-access-f7w27\") pod \"sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc\" (UID: \"477c529e-b2ce-4dca-ab1c-987efbade140\") " pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:24:43.420182 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.420162 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:24:43.678647 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.678575 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="983daa40-2b81-4a89-9e72-a719b5d93eb6" path="/var/lib/kubelet/pods/983daa40-2b81-4a89-9e72-a719b5d93eb6/volumes" Sep 11 01:24:43.690198 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:43.690178 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc"] Sep 11 01:24:43.692480 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:24:43.692458 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod477c529e_b2ce_4dca_ab1c_987efbade140.slice/crio-e95239f51c56ffc4dfb756c07089d78734c91c516a682895590cf55b5da44fee WatchSource:0}: Error finding container e95239f51c56ffc4dfb756c07089d78734c91c516a682895590cf55b5da44fee: Status 404 returned error can't find the container with id e95239f51c56ffc4dfb756c07089d78734c91c516a682895590cf55b5da44fee Sep 11 01:24:44.510044 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:44.510003 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" event={"ID":"477c529e-b2ce-4dca-ab1c-987efbade140","Type":"ContainerStarted","Data":"9ca0de3683383c5b752f267e0f380184902e25bff3d811dc033e07608b6c3a97"} Sep 11 01:24:44.510219 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:44.510062 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" event={"ID":"477c529e-b2ce-4dca-ab1c-987efbade140","Type":"ContainerStarted","Data":"e95239f51c56ffc4dfb756c07089d78734c91c516a682895590cf55b5da44fee"} Sep 11 01:24:44.529471 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:44.529416 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" podStartSLOduration=1.529400573 podStartE2EDuration="1.529400573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:24:44.526983006 +0000 UTC m=+1611.350436518" watchObservedRunningTime="2026-09-11 01:24:44.529400573 +0000 UTC m=+1611.352854082" Sep 11 01:24:44.619261 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:44.619229 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:24:53.420863 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:53.420758 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:24:53.420863 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:53.420804 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:24:53.433499 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:53.433475 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:24:53.563882 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:24:53.563834 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:25:04.002320 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.002284 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr"] Sep 11 01:25:04.006216 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.006190 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:04.008583 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.008554 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9433b9\"/\"openshift-service-ca.crt\"" Sep 11 01:25:04.008732 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.008589 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9433b9\"/\"kube-root-ca.crt\"" Sep 11 01:25:04.009546 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.009523 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9433b9\"/\"default-dockercfg-9w2hq\"" Sep 11 01:25:04.009880 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.009543 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9433b9\"/\"sub-test-model-9433b9-kserve-self-signed-certs\"" Sep 11 01:25:04.017775 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.017749 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr"] Sep 11 01:25:04.043102 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.043070 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:25:04.072044 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.072019 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56dae03c-a52b-4209-b122-36e6a5609fbc-tmp-dir\") pod \"sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr\" (UID: \"56dae03c-a52b-4209-b122-36e6a5609fbc\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:04.072151 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.072085 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56dae03c-a52b-4209-b122-36e6a5609fbc-dshm\") pod \"sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr\" (UID: \"56dae03c-a52b-4209-b122-36e6a5609fbc\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:04.072198 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.072158 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56dae03c-a52b-4209-b122-36e6a5609fbc-model-cache\") pod \"sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr\" (UID: \"56dae03c-a52b-4209-b122-36e6a5609fbc\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:04.072198 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.072188 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqhxp\" (UniqueName: \"kubernetes.io/projected/56dae03c-a52b-4209-b122-36e6a5609fbc-kube-api-access-fqhxp\") pod \"sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr\" (UID: \"56dae03c-a52b-4209-b122-36e6a5609fbc\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:04.072281 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.072220 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56dae03c-a52b-4209-b122-36e6a5609fbc-home\") pod \"sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr\" (UID: \"56dae03c-a52b-4209-b122-36e6a5609fbc\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:04.072321 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.072291 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56dae03c-a52b-4209-b122-36e6a5609fbc-tls-certs\") pod \"sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr\" (UID: \"56dae03c-a52b-4209-b122-36e6a5609fbc\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:04.172997 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.172966 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56dae03c-a52b-4209-b122-36e6a5609fbc-model-cache\") pod \"sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr\" (UID: \"56dae03c-a52b-4209-b122-36e6a5609fbc\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:04.173128 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.173003 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqhxp\" (UniqueName: \"kubernetes.io/projected/56dae03c-a52b-4209-b122-36e6a5609fbc-kube-api-access-fqhxp\") pod \"sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr\" (UID: \"56dae03c-a52b-4209-b122-36e6a5609fbc\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:04.173128 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.173027 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56dae03c-a52b-4209-b122-36e6a5609fbc-home\") pod \"sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr\" (UID: \"56dae03c-a52b-4209-b122-36e6a5609fbc\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:04.173128 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.173063 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56dae03c-a52b-4209-b122-36e6a5609fbc-tls-certs\") pod \"sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr\" (UID: \"56dae03c-a52b-4209-b122-36e6a5609fbc\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:04.173128 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.173090 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56dae03c-a52b-4209-b122-36e6a5609fbc-tmp-dir\") pod \"sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr\" (UID: \"56dae03c-a52b-4209-b122-36e6a5609fbc\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:04.173287 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.173129 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56dae03c-a52b-4209-b122-36e6a5609fbc-dshm\") pod \"sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr\" (UID: \"56dae03c-a52b-4209-b122-36e6a5609fbc\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:04.173422 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.173399 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56dae03c-a52b-4209-b122-36e6a5609fbc-model-cache\") pod \"sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr\" (UID: \"56dae03c-a52b-4209-b122-36e6a5609fbc\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:04.173474 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.173448 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56dae03c-a52b-4209-b122-36e6a5609fbc-home\") pod \"sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr\" (UID: \"56dae03c-a52b-4209-b122-36e6a5609fbc\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:04.173474 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.173464 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56dae03c-a52b-4209-b122-36e6a5609fbc-tmp-dir\") pod \"sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr\" (UID: \"56dae03c-a52b-4209-b122-36e6a5609fbc\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:04.175229 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.175210 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56dae03c-a52b-4209-b122-36e6a5609fbc-dshm\") pod \"sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr\" (UID: \"56dae03c-a52b-4209-b122-36e6a5609fbc\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:04.175526 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.175509 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56dae03c-a52b-4209-b122-36e6a5609fbc-tls-certs\") pod \"sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr\" (UID: \"56dae03c-a52b-4209-b122-36e6a5609fbc\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:04.181258 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.181235 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqhxp\" (UniqueName: \"kubernetes.io/projected/56dae03c-a52b-4209-b122-36e6a5609fbc-kube-api-access-fqhxp\") pod \"sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr\" (UID: \"56dae03c-a52b-4209-b122-36e6a5609fbc\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:04.316552 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.316492 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:04.586597 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.586556 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr"] Sep 11 01:25:04.589258 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:25:04.589229 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56dae03c_a52b_4209_b122_36e6a5609fbc.slice/crio-aecda44f1937829c6cbabcd233ad777d9107b8322a2d11b2fca0fbfc69078d33 WatchSource:0}: Error finding container aecda44f1937829c6cbabcd233ad777d9107b8322a2d11b2fca0fbfc69078d33: Status 404 returned error can't find the container with id aecda44f1937829c6cbabcd233ad777d9107b8322a2d11b2fca0fbfc69078d33 Sep 11 01:25:04.606908 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:04.606882 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" event={"ID":"56dae03c-a52b-4209-b122-36e6a5609fbc","Type":"ContainerStarted","Data":"aecda44f1937829c6cbabcd233ad777d9107b8322a2d11b2fca0fbfc69078d33"} Sep 11 01:25:05.613594 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:05.613554 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" event={"ID":"56dae03c-a52b-4209-b122-36e6a5609fbc","Type":"ContainerStarted","Data":"e1c94926eb961121001229c45857f790dc0c81f38618bc1ee0eabb1930c7568f"} Sep 11 01:25:05.631176 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:05.631132 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" podStartSLOduration=2.631117235 podStartE2EDuration="2.631117235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:25:05.628665462 +0000 UTC m=+1632.452118963" watchObservedRunningTime="2026-09-11 01:25:05.631117235 +0000 UTC m=+1632.454570745" Sep 11 01:25:13.203892 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:13.203855 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:25:14.317227 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:14.317195 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:14.317227 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:14.317237 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:14.329687 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:14.329666 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:14.671533 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:14.671512 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:40.671150 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:40.671114 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc"] Sep 11 01:25:40.687329 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:40.687289 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:25:41.502261 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:41.502225 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr"] Sep 11 01:25:41.526903 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:41.526871 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:25:54.108569 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:54.108531 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:25:55.689553 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:55.689509 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" podUID="477c529e-b2ce-4dca-ab1c-987efbade140" containerName="main" containerID="cri-o://9ca0de3683383c5b752f267e0f380184902e25bff3d811dc033e07608b6c3a97" gracePeriod=45 Sep 11 01:25:55.856306 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:55.856275 2554 generic.go:356] "Generic (PLEG): container finished" podID="477c529e-b2ce-4dca-ab1c-987efbade140" containerID="9ca0de3683383c5b752f267e0f380184902e25bff3d811dc033e07608b6c3a97" exitCode=0 Sep 11 01:25:55.856459 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:55.856329 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" event={"ID":"477c529e-b2ce-4dca-ab1c-987efbade140","Type":"ContainerDied","Data":"9ca0de3683383c5b752f267e0f380184902e25bff3d811dc033e07608b6c3a97"} Sep 11 01:25:55.932775 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:55.932751 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:25:55.995335 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:55.995272 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/477c529e-b2ce-4dca-ab1c-987efbade140-home\") pod \"477c529e-b2ce-4dca-ab1c-987efbade140\" (UID: \"477c529e-b2ce-4dca-ab1c-987efbade140\") " Sep 11 01:25:55.995335 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:55.995302 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/477c529e-b2ce-4dca-ab1c-987efbade140-tmp-dir\") pod \"477c529e-b2ce-4dca-ab1c-987efbade140\" (UID: \"477c529e-b2ce-4dca-ab1c-987efbade140\") " Sep 11 01:25:55.995488 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:55.995349 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/477c529e-b2ce-4dca-ab1c-987efbade140-dshm\") pod \"477c529e-b2ce-4dca-ab1c-987efbade140\" (UID: \"477c529e-b2ce-4dca-ab1c-987efbade140\") " Sep 11 01:25:55.995488 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:55.995388 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/477c529e-b2ce-4dca-ab1c-987efbade140-tls-certs\") pod \"477c529e-b2ce-4dca-ab1c-987efbade140\" (UID: \"477c529e-b2ce-4dca-ab1c-987efbade140\") " Sep 11 01:25:55.995488 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:55.995456 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7w27\" (UniqueName: \"kubernetes.io/projected/477c529e-b2ce-4dca-ab1c-987efbade140-kube-api-access-f7w27\") pod \"477c529e-b2ce-4dca-ab1c-987efbade140\" (UID: \"477c529e-b2ce-4dca-ab1c-987efbade140\") " Sep 11 01:25:55.995602 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:55.995479 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/477c529e-b2ce-4dca-ab1c-987efbade140-home" (OuterVolumeSpecName: "home") pod "477c529e-b2ce-4dca-ab1c-987efbade140" (UID: "477c529e-b2ce-4dca-ab1c-987efbade140"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:25:55.995602 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:55.995497 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/477c529e-b2ce-4dca-ab1c-987efbade140-model-cache\") pod \"477c529e-b2ce-4dca-ab1c-987efbade140\" (UID: \"477c529e-b2ce-4dca-ab1c-987efbade140\") " Sep 11 01:25:55.995696 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:55.995609 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/477c529e-b2ce-4dca-ab1c-987efbade140-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "477c529e-b2ce-4dca-ab1c-987efbade140" (UID: "477c529e-b2ce-4dca-ab1c-987efbade140"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:25:55.995788 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:55.995770 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/477c529e-b2ce-4dca-ab1c-987efbade140-home\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:25:55.995864 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:55.995791 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/477c529e-b2ce-4dca-ab1c-987efbade140-tmp-dir\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:25:55.995864 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:55.995799 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/477c529e-b2ce-4dca-ab1c-987efbade140-model-cache" (OuterVolumeSpecName: "model-cache") pod "477c529e-b2ce-4dca-ab1c-987efbade140" (UID: "477c529e-b2ce-4dca-ab1c-987efbade140"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:25:55.997517 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:55.997496 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/477c529e-b2ce-4dca-ab1c-987efbade140-dshm" (OuterVolumeSpecName: "dshm") pod "477c529e-b2ce-4dca-ab1c-987efbade140" (UID: "477c529e-b2ce-4dca-ab1c-987efbade140"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:25:55.997613 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:55.997534 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/477c529e-b2ce-4dca-ab1c-987efbade140-kube-api-access-f7w27" (OuterVolumeSpecName: "kube-api-access-f7w27") pod "477c529e-b2ce-4dca-ab1c-987efbade140" (UID: "477c529e-b2ce-4dca-ab1c-987efbade140"). InnerVolumeSpecName "kube-api-access-f7w27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:25:55.997613 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:55.997576 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/477c529e-b2ce-4dca-ab1c-987efbade140-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "477c529e-b2ce-4dca-ab1c-987efbade140" (UID: "477c529e-b2ce-4dca-ab1c-987efbade140"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:25:56.096631 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.096611 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/477c529e-b2ce-4dca-ab1c-987efbade140-model-cache\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:25:56.096631 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.096630 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/477c529e-b2ce-4dca-ab1c-987efbade140-dshm\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:25:56.096755 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.096639 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/477c529e-b2ce-4dca-ab1c-987efbade140-tls-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:25:56.096755 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.096649 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7w27\" (UniqueName: \"kubernetes.io/projected/477c529e-b2ce-4dca-ab1c-987efbade140-kube-api-access-f7w27\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:25:56.521713 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.521680 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" podUID="56dae03c-a52b-4209-b122-36e6a5609fbc" containerName="main" containerID="cri-o://e1c94926eb961121001229c45857f790dc0c81f38618bc1ee0eabb1930c7568f" gracePeriod=45 Sep 11 01:25:56.772771 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.772711 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:56.802895 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.802871 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56dae03c-a52b-4209-b122-36e6a5609fbc-tls-certs\") pod \"56dae03c-a52b-4209-b122-36e6a5609fbc\" (UID: \"56dae03c-a52b-4209-b122-36e6a5609fbc\") " Sep 11 01:25:56.803008 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.802919 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56dae03c-a52b-4209-b122-36e6a5609fbc-dshm\") pod \"56dae03c-a52b-4209-b122-36e6a5609fbc\" (UID: \"56dae03c-a52b-4209-b122-36e6a5609fbc\") " Sep 11 01:25:56.803008 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.802945 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56dae03c-a52b-4209-b122-36e6a5609fbc-home\") pod \"56dae03c-a52b-4209-b122-36e6a5609fbc\" (UID: \"56dae03c-a52b-4209-b122-36e6a5609fbc\") " Sep 11 01:25:56.803008 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.802971 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqhxp\" (UniqueName: \"kubernetes.io/projected/56dae03c-a52b-4209-b122-36e6a5609fbc-kube-api-access-fqhxp\") pod \"56dae03c-a52b-4209-b122-36e6a5609fbc\" (UID: \"56dae03c-a52b-4209-b122-36e6a5609fbc\") " Sep 11 01:25:56.803008 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.802995 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56dae03c-a52b-4209-b122-36e6a5609fbc-model-cache\") pod \"56dae03c-a52b-4209-b122-36e6a5609fbc\" (UID: \"56dae03c-a52b-4209-b122-36e6a5609fbc\") " Sep 11 01:25:56.803216 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.803032 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56dae03c-a52b-4209-b122-36e6a5609fbc-tmp-dir\") pod \"56dae03c-a52b-4209-b122-36e6a5609fbc\" (UID: \"56dae03c-a52b-4209-b122-36e6a5609fbc\") " Sep 11 01:25:56.803290 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.803263 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56dae03c-a52b-4209-b122-36e6a5609fbc-model-cache" (OuterVolumeSpecName: "model-cache") pod "56dae03c-a52b-4209-b122-36e6a5609fbc" (UID: "56dae03c-a52b-4209-b122-36e6a5609fbc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:25:56.803360 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.803292 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56dae03c-a52b-4209-b122-36e6a5609fbc-home" (OuterVolumeSpecName: "home") pod "56dae03c-a52b-4209-b122-36e6a5609fbc" (UID: "56dae03c-a52b-4209-b122-36e6a5609fbc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:25:56.803360 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.803311 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56dae03c-a52b-4209-b122-36e6a5609fbc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "56dae03c-a52b-4209-b122-36e6a5609fbc" (UID: "56dae03c-a52b-4209-b122-36e6a5609fbc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:25:56.803590 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.803575 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56dae03c-a52b-4209-b122-36e6a5609fbc-home\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:25:56.803633 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.803596 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56dae03c-a52b-4209-b122-36e6a5609fbc-model-cache\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:25:56.803633 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.803613 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56dae03c-a52b-4209-b122-36e6a5609fbc-tmp-dir\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:25:56.805340 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.805320 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56dae03c-a52b-4209-b122-36e6a5609fbc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56dae03c-a52b-4209-b122-36e6a5609fbc" (UID: "56dae03c-a52b-4209-b122-36e6a5609fbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:25:56.805415 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.805323 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56dae03c-a52b-4209-b122-36e6a5609fbc-dshm" (OuterVolumeSpecName: "dshm") pod "56dae03c-a52b-4209-b122-36e6a5609fbc" (UID: "56dae03c-a52b-4209-b122-36e6a5609fbc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:25:56.805549 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.805529 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56dae03c-a52b-4209-b122-36e6a5609fbc-kube-api-access-fqhxp" (OuterVolumeSpecName: "kube-api-access-fqhxp") pod "56dae03c-a52b-4209-b122-36e6a5609fbc" (UID: "56dae03c-a52b-4209-b122-36e6a5609fbc"). InnerVolumeSpecName "kube-api-access-fqhxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:25:56.863133 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.863105 2554 generic.go:356] "Generic (PLEG): container finished" podID="56dae03c-a52b-4209-b122-36e6a5609fbc" containerID="e1c94926eb961121001229c45857f790dc0c81f38618bc1ee0eabb1930c7568f" exitCode=0 Sep 11 01:25:56.863247 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.863149 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" Sep 11 01:25:56.863247 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.863183 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" event={"ID":"56dae03c-a52b-4209-b122-36e6a5609fbc","Type":"ContainerDied","Data":"e1c94926eb961121001229c45857f790dc0c81f38618bc1ee0eabb1930c7568f"} Sep 11 01:25:56.863247 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.863227 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr" event={"ID":"56dae03c-a52b-4209-b122-36e6a5609fbc","Type":"ContainerDied","Data":"aecda44f1937829c6cbabcd233ad777d9107b8322a2d11b2fca0fbfc69078d33"} Sep 11 01:25:56.863247 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.863245 2554 scope.go:117] "RemoveContainer" containerID="e1c94926eb961121001229c45857f790dc0c81f38618bc1ee0eabb1930c7568f" Sep 11 01:25:56.865007 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.864972 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" Sep 11 01:25:56.865007 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.864966 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc" event={"ID":"477c529e-b2ce-4dca-ab1c-987efbade140","Type":"ContainerDied","Data":"e95239f51c56ffc4dfb756c07089d78734c91c516a682895590cf55b5da44fee"} Sep 11 01:25:56.872804 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.872786 2554 scope.go:117] "RemoveContainer" containerID="e1c94926eb961121001229c45857f790dc0c81f38618bc1ee0eabb1930c7568f" Sep 11 01:25:56.873083 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:25:56.873067 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1c94926eb961121001229c45857f790dc0c81f38618bc1ee0eabb1930c7568f\": container with ID starting with e1c94926eb961121001229c45857f790dc0c81f38618bc1ee0eabb1930c7568f not found: ID does not exist" containerID="e1c94926eb961121001229c45857f790dc0c81f38618bc1ee0eabb1930c7568f" Sep 11 01:25:56.873148 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.873089 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c94926eb961121001229c45857f790dc0c81f38618bc1ee0eabb1930c7568f"} err="failed to get container status \"e1c94926eb961121001229c45857f790dc0c81f38618bc1ee0eabb1930c7568f\": rpc error: code = NotFound desc = could not find container \"e1c94926eb961121001229c45857f790dc0c81f38618bc1ee0eabb1930c7568f\": container with ID starting with e1c94926eb961121001229c45857f790dc0c81f38618bc1ee0eabb1930c7568f not found: ID does not exist" Sep 11 01:25:56.873148 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.873105 2554 scope.go:117] "RemoveContainer" containerID="9ca0de3683383c5b752f267e0f380184902e25bff3d811dc033e07608b6c3a97" Sep 11 01:25:56.884414 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.884395 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr"] Sep 11 01:25:56.888002 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.887982 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9433b9/sub-test-model-9433b9-kserve-85dd8f8df9-cgfwr"] Sep 11 01:25:56.897270 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.897250 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc"] Sep 11 01:25:56.899189 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.899170 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2482ac/sub-test-model-2482ac-kserve-6b9895fcf5-h2nvc"] Sep 11 01:25:56.904355 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.904338 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56dae03c-a52b-4209-b122-36e6a5609fbc-tls-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:25:56.904416 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.904357 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56dae03c-a52b-4209-b122-36e6a5609fbc-dshm\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:25:56.904416 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:56.904366 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqhxp\" (UniqueName: \"kubernetes.io/projected/56dae03c-a52b-4209-b122-36e6a5609fbc-kube-api-access-fqhxp\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:25:57.680206 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:57.680167 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="477c529e-b2ce-4dca-ab1c-987efbade140" path="/var/lib/kubelet/pods/477c529e-b2ce-4dca-ab1c-987efbade140/volumes" Sep 11 01:25:57.680758 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:25:57.680736 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56dae03c-a52b-4209-b122-36e6a5609fbc" path="/var/lib/kubelet/pods/56dae03c-a52b-4209-b122-36e6a5609fbc/volumes" Sep 11 01:26:24.003211 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.003177 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr"] Sep 11 01:26:24.003811 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.003716 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="477c529e-b2ce-4dca-ab1c-987efbade140" containerName="main" Sep 11 01:26:24.003811 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.003737 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="477c529e-b2ce-4dca-ab1c-987efbade140" containerName="main" Sep 11 01:26:24.003811 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.003770 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56dae03c-a52b-4209-b122-36e6a5609fbc" containerName="main" Sep 11 01:26:24.003811 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.003779 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="56dae03c-a52b-4209-b122-36e6a5609fbc" containerName="main" Sep 11 01:26:24.004065 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.003876 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="56dae03c-a52b-4209-b122-36e6a5609fbc" containerName="main" Sep 11 01:26:24.004065 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.003889 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="477c529e-b2ce-4dca-ab1c-987efbade140" containerName="main" Sep 11 01:26:24.007011 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.006987 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:26:24.009391 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.009366 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9433b9\"/\"kube-root-ca.crt\"" Sep 11 01:26:24.010123 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.010099 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9433b9\"/\"rate-test-model-9433b9-kserve-self-signed-certs\"" Sep 11 01:26:24.010269 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.010248 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9433b9\"/\"openshift-service-ca.crt\"" Sep 11 01:26:24.010358 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.010289 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9433b9\"/\"default-dockercfg-9w2hq\"" Sep 11 01:26:24.016097 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.016075 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr"] Sep 11 01:26:24.023131 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.023101 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:26:24.112047 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.112023 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsllt\" (UniqueName: \"kubernetes.io/projected/a97b88ef-60a3-4c49-80da-d32d83e314d4-kube-api-access-xsllt\") pod \"rate-test-model-9433b9-kserve-6cc5f85579-g67jr\" (UID: \"a97b88ef-60a3-4c49-80da-d32d83e314d4\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:26:24.112149 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.112082 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a97b88ef-60a3-4c49-80da-d32d83e314d4-tmp-dir\") pod \"rate-test-model-9433b9-kserve-6cc5f85579-g67jr\" (UID: \"a97b88ef-60a3-4c49-80da-d32d83e314d4\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:26:24.112149 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.112136 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a97b88ef-60a3-4c49-80da-d32d83e314d4-dshm\") pod \"rate-test-model-9433b9-kserve-6cc5f85579-g67jr\" (UID: \"a97b88ef-60a3-4c49-80da-d32d83e314d4\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:26:24.112223 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.112155 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a97b88ef-60a3-4c49-80da-d32d83e314d4-home\") pod \"rate-test-model-9433b9-kserve-6cc5f85579-g67jr\" (UID: \"a97b88ef-60a3-4c49-80da-d32d83e314d4\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:26:24.112223 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.112209 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a97b88ef-60a3-4c49-80da-d32d83e314d4-model-cache\") pod \"rate-test-model-9433b9-kserve-6cc5f85579-g67jr\" (UID: \"a97b88ef-60a3-4c49-80da-d32d83e314d4\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:26:24.112293 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.112249 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a97b88ef-60a3-4c49-80da-d32d83e314d4-tls-certs\") pod \"rate-test-model-9433b9-kserve-6cc5f85579-g67jr\" (UID: \"a97b88ef-60a3-4c49-80da-d32d83e314d4\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:26:24.213145 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.213116 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a97b88ef-60a3-4c49-80da-d32d83e314d4-tmp-dir\") pod \"rate-test-model-9433b9-kserve-6cc5f85579-g67jr\" (UID: \"a97b88ef-60a3-4c49-80da-d32d83e314d4\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:26:24.213252 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.213162 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a97b88ef-60a3-4c49-80da-d32d83e314d4-dshm\") pod \"rate-test-model-9433b9-kserve-6cc5f85579-g67jr\" (UID: \"a97b88ef-60a3-4c49-80da-d32d83e314d4\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:26:24.213252 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.213190 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a97b88ef-60a3-4c49-80da-d32d83e314d4-home\") pod \"rate-test-model-9433b9-kserve-6cc5f85579-g67jr\" (UID: \"a97b88ef-60a3-4c49-80da-d32d83e314d4\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:26:24.213353 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.213217 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a97b88ef-60a3-4c49-80da-d32d83e314d4-model-cache\") pod \"rate-test-model-9433b9-kserve-6cc5f85579-g67jr\" (UID: \"a97b88ef-60a3-4c49-80da-d32d83e314d4\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:26:24.213353 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.213288 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a97b88ef-60a3-4c49-80da-d32d83e314d4-tls-certs\") pod \"rate-test-model-9433b9-kserve-6cc5f85579-g67jr\" (UID: \"a97b88ef-60a3-4c49-80da-d32d83e314d4\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:26:24.213353 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.213324 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsllt\" (UniqueName: \"kubernetes.io/projected/a97b88ef-60a3-4c49-80da-d32d83e314d4-kube-api-access-xsllt\") pod \"rate-test-model-9433b9-kserve-6cc5f85579-g67jr\" (UID: \"a97b88ef-60a3-4c49-80da-d32d83e314d4\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:26:24.213566 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.213546 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a97b88ef-60a3-4c49-80da-d32d83e314d4-tmp-dir\") pod \"rate-test-model-9433b9-kserve-6cc5f85579-g67jr\" (UID: \"a97b88ef-60a3-4c49-80da-d32d83e314d4\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:26:24.213669 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.213627 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a97b88ef-60a3-4c49-80da-d32d83e314d4-model-cache\") pod \"rate-test-model-9433b9-kserve-6cc5f85579-g67jr\" (UID: \"a97b88ef-60a3-4c49-80da-d32d83e314d4\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:26:24.213669 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.213658 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a97b88ef-60a3-4c49-80da-d32d83e314d4-home\") pod \"rate-test-model-9433b9-kserve-6cc5f85579-g67jr\" (UID: \"a97b88ef-60a3-4c49-80da-d32d83e314d4\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:26:24.215455 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.215432 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a97b88ef-60a3-4c49-80da-d32d83e314d4-dshm\") pod \"rate-test-model-9433b9-kserve-6cc5f85579-g67jr\" (UID: \"a97b88ef-60a3-4c49-80da-d32d83e314d4\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:26:24.215697 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.215680 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a97b88ef-60a3-4c49-80da-d32d83e314d4-tls-certs\") pod \"rate-test-model-9433b9-kserve-6cc5f85579-g67jr\" (UID: \"a97b88ef-60a3-4c49-80da-d32d83e314d4\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:26:24.221316 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.221296 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsllt\" (UniqueName: \"kubernetes.io/projected/a97b88ef-60a3-4c49-80da-d32d83e314d4-kube-api-access-xsllt\") pod \"rate-test-model-9433b9-kserve-6cc5f85579-g67jr\" (UID: \"a97b88ef-60a3-4c49-80da-d32d83e314d4\") " pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:26:24.316749 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.316687 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:26:24.593061 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:24.592863 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr"] Sep 11 01:26:24.595229 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:26:24.595207 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda97b88ef_60a3_4c49_80da_d32d83e314d4.slice/crio-7d5133677d6fba9a0d6fc262eb548aaa6c928e6dd1078ebeea0127a4d4fc831f WatchSource:0}: Error finding container 7d5133677d6fba9a0d6fc262eb548aaa6c928e6dd1078ebeea0127a4d4fc831f: Status 404 returned error can't find the container with id 7d5133677d6fba9a0d6fc262eb548aaa6c928e6dd1078ebeea0127a4d4fc831f Sep 11 01:26:25.000928 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:25.000894 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" event={"ID":"a97b88ef-60a3-4c49-80da-d32d83e314d4","Type":"ContainerStarted","Data":"6b832e49d26bb9c0f00ba7b677fbf31527eb3380ee6c39947cf01fdc7c2e5c72"} Sep 11 01:26:25.000928 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:25.000932 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" event={"ID":"a97b88ef-60a3-4c49-80da-d32d83e314d4","Type":"ContainerStarted","Data":"7d5133677d6fba9a0d6fc262eb548aaa6c928e6dd1078ebeea0127a4d4fc831f"} Sep 11 01:26:25.019435 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:25.019387 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" podStartSLOduration=2.019373233 podStartE2EDuration="2.019373233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:26:25.017941724 +0000 UTC m=+1711.841395234" watchObservedRunningTime="2026-09-11 01:26:25.019373233 +0000 UTC m=+1711.842826792" Sep 11 01:26:34.317648 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:34.317615 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:26:34.318038 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:34.317662 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:26:34.330399 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:34.330376 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:26:35.060897 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:35.060864 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:26:49.532916 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:49.532863 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:26:49.589548 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:26:49.589521 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:05.724648 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:05.724613 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:06.712207 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:06.712175 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr"] Sep 11 01:27:06.736253 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:06.736212 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:07.163669 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:07.163572 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:07.469528 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:07.469443 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:07.688904 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:07.688867 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:10.258861 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:10.258804 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:10.391474 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:10.391440 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:11.499945 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:11.499908 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln"] Sep 11 01:27:11.504288 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:11.504261 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" Sep 11 01:27:11.507060 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:11.507041 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-976998-serving-cert\"" Sep 11 01:27:11.514728 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:11.514707 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln"] Sep 11 01:27:11.536018 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:11.535994 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:11.601508 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:11.601486 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9630f40-985d-4037-b549-d237ee02fdad-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-976998-d577768cc-kxzln\" (UID: \"e9630f40-985d-4037-b549-d237ee02fdad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" Sep 11 01:27:11.601631 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:11.601547 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9630f40-985d-4037-b549-d237ee02fdad-maas-api-tls\") pod \"maas-api-e2e-ipp-a-976998-d577768cc-kxzln\" (UID: \"e9630f40-985d-4037-b549-d237ee02fdad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" Sep 11 01:27:11.601631 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:11.601590 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf2nn\" (UniqueName: \"kubernetes.io/projected/e9630f40-985d-4037-b549-d237ee02fdad-kube-api-access-kf2nn\") pod \"maas-api-e2e-ipp-a-976998-d577768cc-kxzln\" (UID: \"e9630f40-985d-4037-b549-d237ee02fdad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" Sep 11 01:27:11.601631 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:11.601620 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9630f40-985d-4037-b549-d237ee02fdad-metrics-certs\") pod \"maas-api-e2e-ipp-a-976998-d577768cc-kxzln\" (UID: \"e9630f40-985d-4037-b549-d237ee02fdad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" Sep 11 01:27:11.639694 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:11.639667 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:11.702024 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:11.701994 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9630f40-985d-4037-b549-d237ee02fdad-maas-api-tls\") pod \"maas-api-e2e-ipp-a-976998-d577768cc-kxzln\" (UID: \"e9630f40-985d-4037-b549-d237ee02fdad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" Sep 11 01:27:11.702142 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:11.702034 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf2nn\" (UniqueName: \"kubernetes.io/projected/e9630f40-985d-4037-b549-d237ee02fdad-kube-api-access-kf2nn\") pod \"maas-api-e2e-ipp-a-976998-d577768cc-kxzln\" (UID: \"e9630f40-985d-4037-b549-d237ee02fdad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" Sep 11 01:27:11.702142 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:11.702089 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9630f40-985d-4037-b549-d237ee02fdad-metrics-certs\") pod \"maas-api-e2e-ipp-a-976998-d577768cc-kxzln\" (UID: \"e9630f40-985d-4037-b549-d237ee02fdad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" Sep 11 01:27:11.702142 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:11.702125 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9630f40-985d-4037-b549-d237ee02fdad-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-976998-d577768cc-kxzln\" (UID: \"e9630f40-985d-4037-b549-d237ee02fdad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" Sep 11 01:27:11.702287 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:27:11.702146 2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-serving-cert: secret "maas-api-e2e-ipp-a-976998-serving-cert" not found Sep 11 01:27:11.702287 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:27:11.702206 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9630f40-985d-4037-b549-d237ee02fdad-maas-api-tls podName:e9630f40-985d-4037-b549-d237ee02fdad nodeName:}" failed. No retries permitted until 2026-09-11 01:27:12.20218761 +0000 UTC m=+1759.025641127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e9630f40-985d-4037-b549-d237ee02fdad-maas-api-tls") pod "maas-api-e2e-ipp-a-976998-d577768cc-kxzln" (UID: "e9630f40-985d-4037-b549-d237ee02fdad") : secret "maas-api-e2e-ipp-a-976998-serving-cert" not found Sep 11 01:27:11.702806 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:11.702788 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9630f40-985d-4037-b549-d237ee02fdad-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-976998-d577768cc-kxzln\" (UID: \"e9630f40-985d-4037-b549-d237ee02fdad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" Sep 11 01:27:11.704705 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:11.704682 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9630f40-985d-4037-b549-d237ee02fdad-metrics-certs\") pod \"maas-api-e2e-ipp-a-976998-d577768cc-kxzln\" (UID: \"e9630f40-985d-4037-b549-d237ee02fdad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" Sep 11 01:27:11.710525 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:11.710499 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf2nn\" (UniqueName: \"kubernetes.io/projected/e9630f40-985d-4037-b549-d237ee02fdad-kube-api-access-kf2nn\") pod \"maas-api-e2e-ipp-a-976998-d577768cc-kxzln\" (UID: \"e9630f40-985d-4037-b549-d237ee02fdad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" Sep 11 01:27:11.776460 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:11.776381 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:12.105174 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:12.105088 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:12.207155 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:12.207112 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:12.207510 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:12.207491 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9630f40-985d-4037-b549-d237ee02fdad-maas-api-tls\") pod \"maas-api-e2e-ipp-a-976998-d577768cc-kxzln\" (UID: \"e9630f40-985d-4037-b549-d237ee02fdad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" Sep 11 01:27:12.209925 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:12.209901 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9630f40-985d-4037-b549-d237ee02fdad-maas-api-tls\") pod \"maas-api-e2e-ipp-a-976998-d577768cc-kxzln\" (UID: \"e9630f40-985d-4037-b549-d237ee02fdad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" Sep 11 01:27:12.419522 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:12.419444 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" Sep 11 01:27:12.692783 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:12.692755 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln"] Sep 11 01:27:12.694497 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:27:12.694471 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9630f40_985d_4037_b549_d237ee02fdad.slice/crio-669768e656a907f20738fc9ecc11a2004f3889bc6e0684b703fe25374086fdb4 WatchSource:0}: Error finding container 669768e656a907f20738fc9ecc11a2004f3889bc6e0684b703fe25374086fdb4: Status 404 returned error can't find the container with id 669768e656a907f20738fc9ecc11a2004f3889bc6e0684b703fe25374086fdb4 Sep 11 01:27:12.695933 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:12.695915 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 01:27:13.230480 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:13.230456 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" event={"ID":"e9630f40-985d-4037-b549-d237ee02fdad","Type":"ContainerStarted","Data":"669768e656a907f20738fc9ecc11a2004f3889bc6e0684b703fe25374086fdb4"} Sep 11 01:27:14.236390 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:14.236354 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" event={"ID":"e9630f40-985d-4037-b549-d237ee02fdad","Type":"ContainerStarted","Data":"9e0522e4cba9536997f9bbc644959ed4ad2a0d89cf625e5308ec601a02e8b055"} Sep 11 01:27:14.236787 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:14.236485 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" Sep 11 01:27:14.253401 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:14.253352 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" podStartSLOduration=2.772749246 podStartE2EDuration="3.253338853s" totalImagesPullingTime="480.589607ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:27:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:27:14.251544786 +0000 UTC m=+1761.074998296" watchObservedRunningTime="2026-09-11 01:27:14.253338853 +0000 UTC m=+1761.076792362" Sep 11 01:27:20.246213 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:20.246173 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" Sep 11 01:27:21.732093 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:21.732059 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" podUID="a97b88ef-60a3-4c49-80da-d32d83e314d4" containerName="main" containerID="cri-o://6b832e49d26bb9c0f00ba7b677fbf31527eb3380ee6c39947cf01fdc7c2e5c72" gracePeriod=45 Sep 11 01:27:21.970883 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:21.970857 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:27:22.087540 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.087469 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a97b88ef-60a3-4c49-80da-d32d83e314d4-tmp-dir\") pod \"a97b88ef-60a3-4c49-80da-d32d83e314d4\" (UID: \"a97b88ef-60a3-4c49-80da-d32d83e314d4\") " Sep 11 01:27:22.087540 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.087518 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a97b88ef-60a3-4c49-80da-d32d83e314d4-home\") pod \"a97b88ef-60a3-4c49-80da-d32d83e314d4\" (UID: \"a97b88ef-60a3-4c49-80da-d32d83e314d4\") " Sep 11 01:27:22.087731 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.087547 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a97b88ef-60a3-4c49-80da-d32d83e314d4-dshm\") pod \"a97b88ef-60a3-4c49-80da-d32d83e314d4\" (UID: \"a97b88ef-60a3-4c49-80da-d32d83e314d4\") " Sep 11 01:27:22.087731 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.087585 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a97b88ef-60a3-4c49-80da-d32d83e314d4-tls-certs\") pod \"a97b88ef-60a3-4c49-80da-d32d83e314d4\" (UID: \"a97b88ef-60a3-4c49-80da-d32d83e314d4\") " Sep 11 01:27:22.087731 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.087644 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a97b88ef-60a3-4c49-80da-d32d83e314d4-model-cache\") pod \"a97b88ef-60a3-4c49-80da-d32d83e314d4\" (UID: \"a97b88ef-60a3-4c49-80da-d32d83e314d4\") " Sep 11 01:27:22.087731 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.087690 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsllt\" (UniqueName: \"kubernetes.io/projected/a97b88ef-60a3-4c49-80da-d32d83e314d4-kube-api-access-xsllt\") pod \"a97b88ef-60a3-4c49-80da-d32d83e314d4\" (UID: \"a97b88ef-60a3-4c49-80da-d32d83e314d4\") " Sep 11 01:27:22.087731 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.087706 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97b88ef-60a3-4c49-80da-d32d83e314d4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a97b88ef-60a3-4c49-80da-d32d83e314d4" (UID: "a97b88ef-60a3-4c49-80da-d32d83e314d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:27:22.088016 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.087754 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97b88ef-60a3-4c49-80da-d32d83e314d4-home" (OuterVolumeSpecName: "home") pod "a97b88ef-60a3-4c49-80da-d32d83e314d4" (UID: "a97b88ef-60a3-4c49-80da-d32d83e314d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:27:22.088016 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.087937 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97b88ef-60a3-4c49-80da-d32d83e314d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "a97b88ef-60a3-4c49-80da-d32d83e314d4" (UID: "a97b88ef-60a3-4c49-80da-d32d83e314d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:27:22.088016 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.087989 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a97b88ef-60a3-4c49-80da-d32d83e314d4-home\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:27:22.088016 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.088008 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a97b88ef-60a3-4c49-80da-d32d83e314d4-tmp-dir\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:27:22.089693 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.089664 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a97b88ef-60a3-4c49-80da-d32d83e314d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a97b88ef-60a3-4c49-80da-d32d83e314d4" (UID: "a97b88ef-60a3-4c49-80da-d32d83e314d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:27:22.089800 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.089757 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97b88ef-60a3-4c49-80da-d32d83e314d4-dshm" (OuterVolumeSpecName: "dshm") pod "a97b88ef-60a3-4c49-80da-d32d83e314d4" (UID: "a97b88ef-60a3-4c49-80da-d32d83e314d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:27:22.089864 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.089794 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a97b88ef-60a3-4c49-80da-d32d83e314d4-kube-api-access-xsllt" (OuterVolumeSpecName: "kube-api-access-xsllt") pod "a97b88ef-60a3-4c49-80da-d32d83e314d4" (UID: "a97b88ef-60a3-4c49-80da-d32d83e314d4"). InnerVolumeSpecName "kube-api-access-xsllt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:27:22.188984 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.188950 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsllt\" (UniqueName: \"kubernetes.io/projected/a97b88ef-60a3-4c49-80da-d32d83e314d4-kube-api-access-xsllt\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:27:22.188984 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.188982 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a97b88ef-60a3-4c49-80da-d32d83e314d4-dshm\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:27:22.188984 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.188993 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a97b88ef-60a3-4c49-80da-d32d83e314d4-tls-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:27:22.189150 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.189002 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a97b88ef-60a3-4c49-80da-d32d83e314d4-model-cache\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:27:22.276260 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.276230 2554 generic.go:356] "Generic (PLEG): container finished" podID="a97b88ef-60a3-4c49-80da-d32d83e314d4" containerID="6b832e49d26bb9c0f00ba7b677fbf31527eb3380ee6c39947cf01fdc7c2e5c72" exitCode=0 Sep 11 01:27:22.276369 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.276276 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" Sep 11 01:27:22.276369 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.276320 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" event={"ID":"a97b88ef-60a3-4c49-80da-d32d83e314d4","Type":"ContainerDied","Data":"6b832e49d26bb9c0f00ba7b677fbf31527eb3380ee6c39947cf01fdc7c2e5c72"} Sep 11 01:27:22.276444 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.276368 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr" event={"ID":"a97b88ef-60a3-4c49-80da-d32d83e314d4","Type":"ContainerDied","Data":"7d5133677d6fba9a0d6fc262eb548aaa6c928e6dd1078ebeea0127a4d4fc831f"} Sep 11 01:27:22.276444 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.276389 2554 scope.go:117] "RemoveContainer" containerID="6b832e49d26bb9c0f00ba7b677fbf31527eb3380ee6c39947cf01fdc7c2e5c72" Sep 11 01:27:22.285569 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.285550 2554 scope.go:117] "RemoveContainer" containerID="6b832e49d26bb9c0f00ba7b677fbf31527eb3380ee6c39947cf01fdc7c2e5c72" Sep 11 01:27:22.285873 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:27:22.285851 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b832e49d26bb9c0f00ba7b677fbf31527eb3380ee6c39947cf01fdc7c2e5c72\": container with ID starting with 6b832e49d26bb9c0f00ba7b677fbf31527eb3380ee6c39947cf01fdc7c2e5c72 not found: ID does not exist" containerID="6b832e49d26bb9c0f00ba7b677fbf31527eb3380ee6c39947cf01fdc7c2e5c72" Sep 11 01:27:22.285917 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.285882 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b832e49d26bb9c0f00ba7b677fbf31527eb3380ee6c39947cf01fdc7c2e5c72"} err="failed to get container status \"6b832e49d26bb9c0f00ba7b677fbf31527eb3380ee6c39947cf01fdc7c2e5c72\": rpc error: code = NotFound desc = could not find container \"6b832e49d26bb9c0f00ba7b677fbf31527eb3380ee6c39947cf01fdc7c2e5c72\": container with ID starting with 6b832e49d26bb9c0f00ba7b677fbf31527eb3380ee6c39947cf01fdc7c2e5c72 not found: ID does not exist" Sep 11 01:27:22.297460 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.297441 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr"] Sep 11 01:27:22.303374 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:22.303354 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9433b9/rate-test-model-9433b9-kserve-6cc5f85579-g67jr"] Sep 11 01:27:23.141167 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:23.141132 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:23.683617 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:23.683584 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a97b88ef-60a3-4c49-80da-d32d83e314d4" path="/var/lib/kubelet/pods/a97b88ef-60a3-4c49-80da-d32d83e314d4/volumes" Sep 11 01:27:33.930728 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:33.930691 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:34.528039 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:34.528001 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:35.305724 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:35.305684 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:36.005227 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:36.005184 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:36.316181 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:36.316081 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:36.608887 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:36.608795 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:39.458933 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:39.458897 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:39.549328 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:39.549294 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:40.558958 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:40.558928 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp"] Sep 11 01:27:40.560078 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:40.560042 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a97b88ef-60a3-4c49-80da-d32d83e314d4" containerName="main" Sep 11 01:27:40.560078 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:40.560067 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a97b88ef-60a3-4c49-80da-d32d83e314d4" containerName="main" Sep 11 01:27:40.560235 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:40.560160 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a97b88ef-60a3-4c49-80da-d32d83e314d4" containerName="main" Sep 11 01:27:40.563891 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:40.563871 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp" Sep 11 01:27:40.566780 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:40.566759 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-bae8d3-serving-cert\"" Sep 11 01:27:40.574889 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:40.574864 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp"] Sep 11 01:27:40.590761 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:40.590733 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:40.653290 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:40.653268 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:40.735587 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:40.735563 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1c56ab8-142c-45f8-950f-19c8f5087473-metrics-certs\") pod \"maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp\" (UID: \"e1c56ab8-142c-45f8-950f-19c8f5087473\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp" Sep 11 01:27:40.735681 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:40.735631 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cqj7\" (UniqueName: \"kubernetes.io/projected/e1c56ab8-142c-45f8-950f-19c8f5087473-kube-api-access-6cqj7\") pod \"maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp\" (UID: \"e1c56ab8-142c-45f8-950f-19c8f5087473\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp" Sep 11 01:27:40.735681 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:40.735663 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1c56ab8-142c-45f8-950f-19c8f5087473-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp\" (UID: \"e1c56ab8-142c-45f8-950f-19c8f5087473\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp" Sep 11 01:27:40.735754 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:40.735691 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1c56ab8-142c-45f8-950f-19c8f5087473-maas-api-tls\") pod \"maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp\" (UID: \"e1c56ab8-142c-45f8-950f-19c8f5087473\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp" Sep 11 01:27:40.834250 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:40.834185 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:40.836321 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:40.836298 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cqj7\" (UniqueName: \"kubernetes.io/projected/e1c56ab8-142c-45f8-950f-19c8f5087473-kube-api-access-6cqj7\") pod \"maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp\" (UID: \"e1c56ab8-142c-45f8-950f-19c8f5087473\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp" Sep 11 01:27:40.836437 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:40.836345 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1c56ab8-142c-45f8-950f-19c8f5087473-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp\" (UID: \"e1c56ab8-142c-45f8-950f-19c8f5087473\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp" Sep 11 01:27:40.836437 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:40.836387 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1c56ab8-142c-45f8-950f-19c8f5087473-maas-api-tls\") pod \"maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp\" (UID: \"e1c56ab8-142c-45f8-950f-19c8f5087473\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp" Sep 11 01:27:40.836437 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:40.836419 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1c56ab8-142c-45f8-950f-19c8f5087473-metrics-certs\") pod \"maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp\" (UID: \"e1c56ab8-142c-45f8-950f-19c8f5087473\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp" Sep 11 01:27:40.837099 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:40.837070 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1c56ab8-142c-45f8-950f-19c8f5087473-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp\" (UID: \"e1c56ab8-142c-45f8-950f-19c8f5087473\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp" Sep 11 01:27:40.838898 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:40.838876 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1c56ab8-142c-45f8-950f-19c8f5087473-metrics-certs\") pod \"maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp\" (UID: \"e1c56ab8-142c-45f8-950f-19c8f5087473\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp" Sep 11 01:27:40.839095 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:40.839076 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1c56ab8-142c-45f8-950f-19c8f5087473-maas-api-tls\") pod \"maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp\" (UID: \"e1c56ab8-142c-45f8-950f-19c8f5087473\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp" Sep 11 01:27:40.844330 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:40.844309 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cqj7\" (UniqueName: \"kubernetes.io/projected/e1c56ab8-142c-45f8-950f-19c8f5087473-kube-api-access-6cqj7\") pod \"maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp\" (UID: \"e1c56ab8-142c-45f8-950f-19c8f5087473\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp" Sep 11 01:27:40.878337 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:40.878316 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp" Sep 11 01:27:41.151111 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:41.151020 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp"] Sep 11 01:27:41.154029 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:27:41.154000 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c56ab8_142c_45f8_950f_19c8f5087473.slice/crio-06aaf7fb81d2ede99bf05ed63b2a60aff0d965a3bb3ed538ccef141acf38583f WatchSource:0}: Error finding container 06aaf7fb81d2ede99bf05ed63b2a60aff0d965a3bb3ed538ccef141acf38583f: Status 404 returned error can't find the container with id 06aaf7fb81d2ede99bf05ed63b2a60aff0d965a3bb3ed538ccef141acf38583f Sep 11 01:27:41.179473 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:41.179445 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:41.276280 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:41.276251 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:27:41.367216 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:41.367180 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp" event={"ID":"e1c56ab8-142c-45f8-950f-19c8f5087473","Type":"ContainerStarted","Data":"06aaf7fb81d2ede99bf05ed63b2a60aff0d965a3bb3ed538ccef141acf38583f"} Sep 11 01:27:42.373898 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:42.373863 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp" event={"ID":"e1c56ab8-142c-45f8-950f-19c8f5087473","Type":"ContainerStarted","Data":"9a9bafe0c92285c9330d74940c5a2b56b8ff1894e1575527474a25d841992d1d"} Sep 11 01:27:42.374291 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:42.373931 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp" Sep 11 01:27:42.392912 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:42.392865 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp" podStartSLOduration=2.06736146 podStartE2EDuration="2.392827692s" totalImagesPullingTime="325.466232ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:27:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:27:42.391307226 +0000 UTC m=+1789.214760733" watchObservedRunningTime="2026-09-11 01:27:42.392827692 +0000 UTC m=+1789.216281198" Sep 11 01:27:48.384268 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:48.384192 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp" Sep 11 01:27:53.724034 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:53.724006 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wtd8v_c2b320dd-43e9-435d-8e4f-2bca1f1233ff/ovn-acl-logging/0.log" Sep 11 01:27:53.727291 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:27:53.727271 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wtd8v_c2b320dd-43e9-435d-8e4f-2bca1f1233ff/ovn-acl-logging/0.log" Sep 11 01:28:04.441984 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:04.441945 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:28:10.361780 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:10.361745 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:28:10.463382 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:10.463354 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:28:19.095382 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.095345 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9"] Sep 11 01:28:19.098790 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.098769 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:28:19.101385 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.101362 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-976998\"/\"kube-root-ca.crt\"" Sep 11 01:28:19.101500 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.101361 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-976998\"/\"openshift-service-ca.crt\"" Sep 11 01:28:19.102658 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.102642 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-976998\"/\"default-dockercfg-cswrm\"" Sep 11 01:28:19.102733 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.102662 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-976998\"/\"ipp-route-976998-kserve-self-signed-certs\"" Sep 11 01:28:19.110556 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.110535 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9"] Sep 11 01:28:19.129627 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.129601 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:28:19.242165 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.242137 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff83015b-8ad2-42da-9725-43905db25ee0-tmp-dir\") pod \"ipp-route-976998-kserve-67d77cb779-qdmb9\" (UID: \"ff83015b-8ad2-42da-9725-43905db25ee0\") " pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:28:19.242286 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.242181 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff83015b-8ad2-42da-9725-43905db25ee0-dshm\") pod \"ipp-route-976998-kserve-67d77cb779-qdmb9\" (UID: \"ff83015b-8ad2-42da-9725-43905db25ee0\") " pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:28:19.242286 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.242268 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff83015b-8ad2-42da-9725-43905db25ee0-tls-certs\") pod \"ipp-route-976998-kserve-67d77cb779-qdmb9\" (UID: \"ff83015b-8ad2-42da-9725-43905db25ee0\") " pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:28:19.242377 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.242346 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68vh4\" (UniqueName: \"kubernetes.io/projected/ff83015b-8ad2-42da-9725-43905db25ee0-kube-api-access-68vh4\") pod \"ipp-route-976998-kserve-67d77cb779-qdmb9\" (UID: \"ff83015b-8ad2-42da-9725-43905db25ee0\") " pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:28:19.242444 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.242400 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff83015b-8ad2-42da-9725-43905db25ee0-home\") pod \"ipp-route-976998-kserve-67d77cb779-qdmb9\" (UID: \"ff83015b-8ad2-42da-9725-43905db25ee0\") " pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:28:19.242501 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.242446 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff83015b-8ad2-42da-9725-43905db25ee0-model-cache\") pod \"ipp-route-976998-kserve-67d77cb779-qdmb9\" (UID: \"ff83015b-8ad2-42da-9725-43905db25ee0\") " pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:28:19.343994 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.343958 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff83015b-8ad2-42da-9725-43905db25ee0-tls-certs\") pod \"ipp-route-976998-kserve-67d77cb779-qdmb9\" (UID: \"ff83015b-8ad2-42da-9725-43905db25ee0\") " pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:28:19.344141 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.344035 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68vh4\" (UniqueName: \"kubernetes.io/projected/ff83015b-8ad2-42da-9725-43905db25ee0-kube-api-access-68vh4\") pod \"ipp-route-976998-kserve-67d77cb779-qdmb9\" (UID: \"ff83015b-8ad2-42da-9725-43905db25ee0\") " pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:28:19.344141 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.344082 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff83015b-8ad2-42da-9725-43905db25ee0-home\") pod \"ipp-route-976998-kserve-67d77cb779-qdmb9\" (UID: \"ff83015b-8ad2-42da-9725-43905db25ee0\") " pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:28:19.344249 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.344138 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff83015b-8ad2-42da-9725-43905db25ee0-model-cache\") pod \"ipp-route-976998-kserve-67d77cb779-qdmb9\" (UID: \"ff83015b-8ad2-42da-9725-43905db25ee0\") " pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:28:19.344249 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.344205 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff83015b-8ad2-42da-9725-43905db25ee0-tmp-dir\") pod \"ipp-route-976998-kserve-67d77cb779-qdmb9\" (UID: \"ff83015b-8ad2-42da-9725-43905db25ee0\") " pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:28:19.344249 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.344245 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff83015b-8ad2-42da-9725-43905db25ee0-dshm\") pod \"ipp-route-976998-kserve-67d77cb779-qdmb9\" (UID: \"ff83015b-8ad2-42da-9725-43905db25ee0\") " pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:28:19.344557 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.344516 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff83015b-8ad2-42da-9725-43905db25ee0-home\") pod \"ipp-route-976998-kserve-67d77cb779-qdmb9\" (UID: \"ff83015b-8ad2-42da-9725-43905db25ee0\") " pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:28:19.344721 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.344676 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff83015b-8ad2-42da-9725-43905db25ee0-model-cache\") pod \"ipp-route-976998-kserve-67d77cb779-qdmb9\" (UID: \"ff83015b-8ad2-42da-9725-43905db25ee0\") " pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:28:19.344925 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.344903 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff83015b-8ad2-42da-9725-43905db25ee0-tmp-dir\") pod \"ipp-route-976998-kserve-67d77cb779-qdmb9\" (UID: \"ff83015b-8ad2-42da-9725-43905db25ee0\") " pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:28:19.346899 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.346812 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff83015b-8ad2-42da-9725-43905db25ee0-dshm\") pod \"ipp-route-976998-kserve-67d77cb779-qdmb9\" (UID: \"ff83015b-8ad2-42da-9725-43905db25ee0\") " pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:28:19.347211 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.347192 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff83015b-8ad2-42da-9725-43905db25ee0-tls-certs\") pod \"ipp-route-976998-kserve-67d77cb779-qdmb9\" (UID: \"ff83015b-8ad2-42da-9725-43905db25ee0\") " pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:28:19.353808 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.353789 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68vh4\" (UniqueName: \"kubernetes.io/projected/ff83015b-8ad2-42da-9725-43905db25ee0-kube-api-access-68vh4\") pod \"ipp-route-976998-kserve-67d77cb779-qdmb9\" (UID: \"ff83015b-8ad2-42da-9725-43905db25ee0\") " pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:28:19.409453 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.409426 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:28:19.684256 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:19.684231 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9"] Sep 11 01:28:19.686209 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:28:19.686182 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff83015b_8ad2_42da_9725_43905db25ee0.slice/crio-5aca46501a0c1ce2edbf11b34f7133a9cfa11f68668ed78203c04ab1b351451c WatchSource:0}: Error finding container 5aca46501a0c1ce2edbf11b34f7133a9cfa11f68668ed78203c04ab1b351451c: Status 404 returned error can't find the container with id 5aca46501a0c1ce2edbf11b34f7133a9cfa11f68668ed78203c04ab1b351451c Sep 11 01:28:20.561083 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:20.561046 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" event={"ID":"ff83015b-8ad2-42da-9725-43905db25ee0","Type":"ContainerStarted","Data":"e6d7e13de9bedefb7314948fd4d7cde8b28ed15255ceb3cdcba6a2f203c96fc8"} Sep 11 01:28:20.561453 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:20.561096 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" event={"ID":"ff83015b-8ad2-42da-9725-43905db25ee0","Type":"ContainerStarted","Data":"5aca46501a0c1ce2edbf11b34f7133a9cfa11f68668ed78203c04ab1b351451c"} Sep 11 01:28:20.577912 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:20.577870 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" podStartSLOduration=1.5778553130000001 podStartE2EDuration="1.577855313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:28:20.576031296 +0000 UTC m=+1827.399484808" watchObservedRunningTime="2026-09-11 01:28:20.577855313 +0000 UTC m=+1827.401308814" Sep 11 01:28:29.409902 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:29.409868 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:28:29.410308 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:29.409912 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:28:29.422451 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:29.422425 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:28:29.616658 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:29.616630 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:28:56.328488 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:56.328456 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:28:57.438654 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:28:57.438616 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:00.232706 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:00.232668 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:00.330466 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:00.330429 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:01.500524 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:01.500490 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm"] Sep 11 01:29:01.504470 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:01.504449 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm" Sep 11 01:29:01.506973 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:01.506953 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-40e6bb-serving-cert\"" Sep 11 01:29:01.514895 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:01.514864 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm"] Sep 11 01:29:01.536069 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:01.536044 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:01.592012 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:01.591989 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94ldb\" (UniqueName: \"kubernetes.io/projected/adaf4018-5af7-49f5-b24d-1356fca443b9-kube-api-access-94ldb\") pod \"maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm\" (UID: \"adaf4018-5af7-49f5-b24d-1356fca443b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm" Sep 11 01:29:01.592108 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:01.592029 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adaf4018-5af7-49f5-b24d-1356fca443b9-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm\" (UID: \"adaf4018-5af7-49f5-b24d-1356fca443b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm" Sep 11 01:29:01.592158 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:01.592107 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adaf4018-5af7-49f5-b24d-1356fca443b9-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm\" (UID: \"adaf4018-5af7-49f5-b24d-1356fca443b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm" Sep 11 01:29:01.592158 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:01.592141 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adaf4018-5af7-49f5-b24d-1356fca443b9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm\" (UID: \"adaf4018-5af7-49f5-b24d-1356fca443b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm" Sep 11 01:29:01.596213 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:01.596191 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:01.693208 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:01.693184 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adaf4018-5af7-49f5-b24d-1356fca443b9-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm\" (UID: \"adaf4018-5af7-49f5-b24d-1356fca443b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm" Sep 11 01:29:01.693327 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:01.693225 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adaf4018-5af7-49f5-b24d-1356fca443b9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm\" (UID: \"adaf4018-5af7-49f5-b24d-1356fca443b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm" Sep 11 01:29:01.693327 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:01.693275 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94ldb\" (UniqueName: \"kubernetes.io/projected/adaf4018-5af7-49f5-b24d-1356fca443b9-kube-api-access-94ldb\") pod \"maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm\" (UID: \"adaf4018-5af7-49f5-b24d-1356fca443b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm" Sep 11 01:29:01.693327 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:01.693301 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adaf4018-5af7-49f5-b24d-1356fca443b9-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm\" (UID: \"adaf4018-5af7-49f5-b24d-1356fca443b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm" Sep 11 01:29:01.693936 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:01.693918 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adaf4018-5af7-49f5-b24d-1356fca443b9-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm\" (UID: \"adaf4018-5af7-49f5-b24d-1356fca443b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm" Sep 11 01:29:01.695743 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:01.695726 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adaf4018-5af7-49f5-b24d-1356fca443b9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm\" (UID: \"adaf4018-5af7-49f5-b24d-1356fca443b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm" Sep 11 01:29:01.695908 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:01.695887 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adaf4018-5af7-49f5-b24d-1356fca443b9-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm\" (UID: \"adaf4018-5af7-49f5-b24d-1356fca443b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm" Sep 11 01:29:01.701070 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:01.701049 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94ldb\" (UniqueName: \"kubernetes.io/projected/adaf4018-5af7-49f5-b24d-1356fca443b9-kube-api-access-94ldb\") pod \"maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm\" (UID: \"adaf4018-5af7-49f5-b24d-1356fca443b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm" Sep 11 01:29:01.728647 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:01.728621 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:01.817415 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:01.817350 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm" Sep 11 01:29:01.999910 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:01.996726 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:02.090099 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:02.090074 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm"] Sep 11 01:29:02.091781 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:29:02.091743 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadaf4018_5af7_49f5_b24d_1356fca443b9.slice/crio-808fceb152961430c687c37e9bc363e0aab532ad0e9993e317ba59cd4638e337 WatchSource:0}: Error finding container 808fceb152961430c687c37e9bc363e0aab532ad0e9993e317ba59cd4638e337: Status 404 returned error can't find the container with id 808fceb152961430c687c37e9bc363e0aab532ad0e9993e317ba59cd4638e337 Sep 11 01:29:02.092556 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:02.092435 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:02.769253 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:02.769175 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm" event={"ID":"adaf4018-5af7-49f5-b24d-1356fca443b9","Type":"ContainerStarted","Data":"e46af015b75dee0bc7e99f4ea8f1e4815080df321e9d87c2f008fdeecdb9bc0f"} Sep 11 01:29:02.769253 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:02.769233 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm" event={"ID":"adaf4018-5af7-49f5-b24d-1356fca443b9","Type":"ContainerStarted","Data":"808fceb152961430c687c37e9bc363e0aab532ad0e9993e317ba59cd4638e337"} Sep 11 01:29:02.769623 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:02.769255 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm" Sep 11 01:29:02.789588 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:02.789542 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm" podStartSLOduration=1.414173644 podStartE2EDuration="1.789528177s" totalImagesPullingTime="375.354533ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:29:02.78848773 +0000 UTC m=+1869.611941243" watchObservedRunningTime="2026-09-11 01:29:02.789528177 +0000 UTC m=+1869.612981686" Sep 11 01:29:08.779711 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:08.779680 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm" Sep 11 01:29:25.157579 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:25.157535 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:31.992725 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:31.992690 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm"] Sep 11 01:29:31.993324 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:31.992954 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm" podUID="adaf4018-5af7-49f5-b24d-1356fca443b9" containerName="maas-api" containerID="cri-o://e46af015b75dee0bc7e99f4ea8f1e4815080df321e9d87c2f008fdeecdb9bc0f" gracePeriod=30 Sep 11 01:29:32.023782 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.023713 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:32.092287 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.092251 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:32.158758 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.158666 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:32.224185 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.224146 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:32.238734 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.238714 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm" Sep 11 01:29:32.342412 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.342339 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adaf4018-5af7-49f5-b24d-1356fca443b9-metrics-certs\") pod \"adaf4018-5af7-49f5-b24d-1356fca443b9\" (UID: \"adaf4018-5af7-49f5-b24d-1356fca443b9\") " Sep 11 01:29:32.342412 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.342394 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adaf4018-5af7-49f5-b24d-1356fca443b9-maas-api-tls\") pod \"adaf4018-5af7-49f5-b24d-1356fca443b9\" (UID: \"adaf4018-5af7-49f5-b24d-1356fca443b9\") " Sep 11 01:29:32.342591 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.342445 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94ldb\" (UniqueName: \"kubernetes.io/projected/adaf4018-5af7-49f5-b24d-1356fca443b9-kube-api-access-94ldb\") pod \"adaf4018-5af7-49f5-b24d-1356fca443b9\" (UID: \"adaf4018-5af7-49f5-b24d-1356fca443b9\") " Sep 11 01:29:32.342591 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.342503 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adaf4018-5af7-49f5-b24d-1356fca443b9-openshift-service-ca\") pod \"adaf4018-5af7-49f5-b24d-1356fca443b9\" (UID: \"adaf4018-5af7-49f5-b24d-1356fca443b9\") " Sep 11 01:29:32.342977 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.342941 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adaf4018-5af7-49f5-b24d-1356fca443b9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "adaf4018-5af7-49f5-b24d-1356fca443b9" (UID: "adaf4018-5af7-49f5-b24d-1356fca443b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:29:32.345099 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.345066 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adaf4018-5af7-49f5-b24d-1356fca443b9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "adaf4018-5af7-49f5-b24d-1356fca443b9" (UID: "adaf4018-5af7-49f5-b24d-1356fca443b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:29:32.345235 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.345203 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adaf4018-5af7-49f5-b24d-1356fca443b9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "adaf4018-5af7-49f5-b24d-1356fca443b9" (UID: "adaf4018-5af7-49f5-b24d-1356fca443b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:29:32.345235 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.345206 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adaf4018-5af7-49f5-b24d-1356fca443b9-kube-api-access-94ldb" (OuterVolumeSpecName: "kube-api-access-94ldb") pod "adaf4018-5af7-49f5-b24d-1356fca443b9" (UID: "adaf4018-5af7-49f5-b24d-1356fca443b9"). InnerVolumeSpecName "kube-api-access-94ldb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:29:32.444043 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.444017 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adaf4018-5af7-49f5-b24d-1356fca443b9-metrics-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:29:32.444043 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.444041 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adaf4018-5af7-49f5-b24d-1356fca443b9-maas-api-tls\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:29:32.444189 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.444053 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94ldb\" (UniqueName: \"kubernetes.io/projected/adaf4018-5af7-49f5-b24d-1356fca443b9-kube-api-access-94ldb\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:29:32.444189 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.444067 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adaf4018-5af7-49f5-b24d-1356fca443b9-openshift-service-ca\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:29:32.574216 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.574189 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:32.919757 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.919724 2554 generic.go:356] "Generic (PLEG): container finished" podID="adaf4018-5af7-49f5-b24d-1356fca443b9" containerID="e46af015b75dee0bc7e99f4ea8f1e4815080df321e9d87c2f008fdeecdb9bc0f" exitCode=0 Sep 11 01:29:32.919936 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.919780 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm" Sep 11 01:29:32.919936 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.919804 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm" event={"ID":"adaf4018-5af7-49f5-b24d-1356fca443b9","Type":"ContainerDied","Data":"e46af015b75dee0bc7e99f4ea8f1e4815080df321e9d87c2f008fdeecdb9bc0f"} Sep 11 01:29:32.919936 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.919869 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm" event={"ID":"adaf4018-5af7-49f5-b24d-1356fca443b9","Type":"ContainerDied","Data":"808fceb152961430c687c37e9bc363e0aab532ad0e9993e317ba59cd4638e337"} Sep 11 01:29:32.919936 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.919894 2554 scope.go:117] "RemoveContainer" containerID="e46af015b75dee0bc7e99f4ea8f1e4815080df321e9d87c2f008fdeecdb9bc0f" Sep 11 01:29:32.928700 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.928683 2554 scope.go:117] "RemoveContainer" containerID="e46af015b75dee0bc7e99f4ea8f1e4815080df321e9d87c2f008fdeecdb9bc0f" Sep 11 01:29:32.928989 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:29:32.928971 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e46af015b75dee0bc7e99f4ea8f1e4815080df321e9d87c2f008fdeecdb9bc0f\": container with ID starting with e46af015b75dee0bc7e99f4ea8f1e4815080df321e9d87c2f008fdeecdb9bc0f not found: ID does not exist" containerID="e46af015b75dee0bc7e99f4ea8f1e4815080df321e9d87c2f008fdeecdb9bc0f" Sep 11 01:29:32.929054 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.928996 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e46af015b75dee0bc7e99f4ea8f1e4815080df321e9d87c2f008fdeecdb9bc0f"} err="failed to get container status \"e46af015b75dee0bc7e99f4ea8f1e4815080df321e9d87c2f008fdeecdb9bc0f\": rpc error: code = NotFound desc = could not find container \"e46af015b75dee0bc7e99f4ea8f1e4815080df321e9d87c2f008fdeecdb9bc0f\": container with ID starting with e46af015b75dee0bc7e99f4ea8f1e4815080df321e9d87c2f008fdeecdb9bc0f not found: ID does not exist" Sep 11 01:29:32.940175 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.940153 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm"] Sep 11 01:29:32.946271 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:32.946248 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-40e6bb-6b4d8999c5-85phm"] Sep 11 01:29:33.678724 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:33.678693 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adaf4018-5af7-49f5-b24d-1356fca443b9" path="/var/lib/kubelet/pods/adaf4018-5af7-49f5-b24d-1356fca443b9/volumes" Sep 11 01:29:38.269444 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:38.269404 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:38.369878 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:38.369832 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:38.522283 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:38.522200 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:42.439995 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:42.439957 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:45.275880 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:45.275822 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:50.469558 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:50.469523 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:52.426467 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:52.426426 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:57.977490 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:57.977454 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln"] Sep 11 01:29:57.977978 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:57.977717 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" podUID="e9630f40-985d-4037-b549-d237ee02fdad" containerName="maas-api" containerID="cri-o://9e0522e4cba9536997f9bbc644959ed4ad2a0d89cf625e5308ec601a02e8b055" gracePeriod=30 Sep 11 01:29:58.011489 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:58.011400 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:58.083222 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:58.083192 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:58.159925 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:58.159893 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:58.220800 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:58.220774 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:58.234443 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:58.234390 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" Sep 11 01:29:58.255385 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:58.255364 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kf2nn\" (UniqueName: \"kubernetes.io/projected/e9630f40-985d-4037-b549-d237ee02fdad-kube-api-access-kf2nn\") pod \"e9630f40-985d-4037-b549-d237ee02fdad\" (UID: \"e9630f40-985d-4037-b549-d237ee02fdad\") " Sep 11 01:29:58.255500 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:58.255433 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9630f40-985d-4037-b549-d237ee02fdad-openshift-service-ca\") pod \"e9630f40-985d-4037-b549-d237ee02fdad\" (UID: \"e9630f40-985d-4037-b549-d237ee02fdad\") " Sep 11 01:29:58.255500 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:58.255481 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9630f40-985d-4037-b549-d237ee02fdad-metrics-certs\") pod \"e9630f40-985d-4037-b549-d237ee02fdad\" (UID: \"e9630f40-985d-4037-b549-d237ee02fdad\") " Sep 11 01:29:58.255625 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:58.255545 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9630f40-985d-4037-b549-d237ee02fdad-maas-api-tls\") pod \"e9630f40-985d-4037-b549-d237ee02fdad\" (UID: \"e9630f40-985d-4037-b549-d237ee02fdad\") " Sep 11 01:29:58.256172 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:58.256134 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9630f40-985d-4037-b549-d237ee02fdad-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e9630f40-985d-4037-b549-d237ee02fdad" (UID: "e9630f40-985d-4037-b549-d237ee02fdad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:29:58.258066 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:58.258037 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9630f40-985d-4037-b549-d237ee02fdad-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e9630f40-985d-4037-b549-d237ee02fdad" (UID: "e9630f40-985d-4037-b549-d237ee02fdad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:29:58.258169 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:58.258046 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9630f40-985d-4037-b549-d237ee02fdad-kube-api-access-kf2nn" (OuterVolumeSpecName: "kube-api-access-kf2nn") pod "e9630f40-985d-4037-b549-d237ee02fdad" (UID: "e9630f40-985d-4037-b549-d237ee02fdad"). InnerVolumeSpecName "kube-api-access-kf2nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:29:58.258214 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:58.258187 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9630f40-985d-4037-b549-d237ee02fdad-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e9630f40-985d-4037-b549-d237ee02fdad" (UID: "e9630f40-985d-4037-b549-d237ee02fdad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:29:58.357105 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:58.357075 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9630f40-985d-4037-b549-d237ee02fdad-metrics-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:29:58.357215 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:58.357108 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9630f40-985d-4037-b549-d237ee02fdad-maas-api-tls\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:29:58.357215 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:58.357124 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf2nn\" (UniqueName: \"kubernetes.io/projected/e9630f40-985d-4037-b549-d237ee02fdad-kube-api-access-kf2nn\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:29:58.357215 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:58.357136 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9630f40-985d-4037-b549-d237ee02fdad-openshift-service-ca\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:29:58.585433 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:58.585365 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:29:59.046194 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:59.046162 2554 generic.go:356] "Generic (PLEG): container finished" podID="e9630f40-985d-4037-b549-d237ee02fdad" containerID="9e0522e4cba9536997f9bbc644959ed4ad2a0d89cf625e5308ec601a02e8b055" exitCode=0 Sep 11 01:29:59.046647 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:59.046211 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" Sep 11 01:29:59.046647 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:59.046229 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" event={"ID":"e9630f40-985d-4037-b549-d237ee02fdad","Type":"ContainerDied","Data":"9e0522e4cba9536997f9bbc644959ed4ad2a0d89cf625e5308ec601a02e8b055"} Sep 11 01:29:59.046647 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:59.046269 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln" event={"ID":"e9630f40-985d-4037-b549-d237ee02fdad","Type":"ContainerDied","Data":"669768e656a907f20738fc9ecc11a2004f3889bc6e0684b703fe25374086fdb4"} Sep 11 01:29:59.046647 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:59.046291 2554 scope.go:117] "RemoveContainer" containerID="9e0522e4cba9536997f9bbc644959ed4ad2a0d89cf625e5308ec601a02e8b055" Sep 11 01:29:59.055663 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:59.055645 2554 scope.go:117] "RemoveContainer" containerID="9e0522e4cba9536997f9bbc644959ed4ad2a0d89cf625e5308ec601a02e8b055" Sep 11 01:29:59.055910 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:29:59.055890 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0522e4cba9536997f9bbc644959ed4ad2a0d89cf625e5308ec601a02e8b055\": container with ID starting with 9e0522e4cba9536997f9bbc644959ed4ad2a0d89cf625e5308ec601a02e8b055 not found: ID does not exist" containerID="9e0522e4cba9536997f9bbc644959ed4ad2a0d89cf625e5308ec601a02e8b055" Sep 11 01:29:59.055976 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:59.055918 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0522e4cba9536997f9bbc644959ed4ad2a0d89cf625e5308ec601a02e8b055"} err="failed to get container status \"9e0522e4cba9536997f9bbc644959ed4ad2a0d89cf625e5308ec601a02e8b055\": rpc error: code = NotFound desc = could not find container \"9e0522e4cba9536997f9bbc644959ed4ad2a0d89cf625e5308ec601a02e8b055\": container with ID starting with 9e0522e4cba9536997f9bbc644959ed4ad2a0d89cf625e5308ec601a02e8b055 not found: ID does not exist" Sep 11 01:29:59.067094 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:59.067071 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln"] Sep 11 01:29:59.069085 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:59.069062 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-976998-d577768cc-kxzln"] Sep 11 01:29:59.677901 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:29:59.677871 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9630f40-985d-4037-b549-d237ee02fdad" path="/var/lib/kubelet/pods/e9630f40-985d-4037-b549-d237ee02fdad/volumes" Sep 11 01:30:00.152547 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:00.152469 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bae8d3-29818170-v9hp5"] Sep 11 01:30:00.153007 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:00.152990 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9630f40-985d-4037-b549-d237ee02fdad" containerName="maas-api" Sep 11 01:30:00.153055 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:00.153008 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9630f40-985d-4037-b549-d237ee02fdad" containerName="maas-api" Sep 11 01:30:00.153055 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:00.153029 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adaf4018-5af7-49f5-b24d-1356fca443b9" containerName="maas-api" Sep 11 01:30:00.153055 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:00.153034 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="adaf4018-5af7-49f5-b24d-1356fca443b9" containerName="maas-api" Sep 11 01:30:00.153145 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:00.153089 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="adaf4018-5af7-49f5-b24d-1356fca443b9" containerName="maas-api" Sep 11 01:30:00.153145 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:00.153098 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9630f40-985d-4037-b549-d237ee02fdad" containerName="maas-api" Sep 11 01:30:00.157490 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:00.157469 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bae8d3-29818170-v9hp5" Sep 11 01:30:00.160193 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:00.160172 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pppkt\"" Sep 11 01:30:00.167807 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:00.167790 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bae8d3-29818170-v9hp5" Sep 11 01:30:00.180884 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:00.180862 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bae8d3-29818170-v9hp5"] Sep 11 01:30:00.442425 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:00.442398 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bae8d3-29818170-v9hp5"] Sep 11 01:30:00.445099 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:30:00.445069 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf20ce962_77ef_4451_bf8c_fcfaa2995c01.slice/crio-32410a5677716997152195b9d258722207865b1848ce5e24177d817f910cb3c2 WatchSource:0}: Error finding container 32410a5677716997152195b9d258722207865b1848ce5e24177d817f910cb3c2: Status 404 returned error can't find the container with id 32410a5677716997152195b9d258722207865b1848ce5e24177d817f910cb3c2 Sep 11 01:30:01.065615 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:01.065583 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bae8d3-29818170-v9hp5" event={"ID":"f20ce962-77ef-4451-bf8c-fcfaa2995c01","Type":"ContainerStarted","Data":"32410a5677716997152195b9d258722207865b1848ce5e24177d817f910cb3c2"} Sep 11 01:30:03.562580 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:03.562499 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9"] Sep 11 01:30:03.828268 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:03.828191 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:30:04.082638 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:04.082555 2554 generic.go:356] "Generic (PLEG): container finished" podID="f20ce962-77ef-4451-bf8c-fcfaa2995c01" containerID="33673718c25651313269cb4ba2218c1f874db2e69d8ac697ab41ff045b1d4873" exitCode=0 Sep 11 01:30:04.082638 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:04.082620 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bae8d3-29818170-v9hp5" event={"ID":"f20ce962-77ef-4451-bf8c-fcfaa2995c01","Type":"ContainerDied","Data":"33673718c25651313269cb4ba2218c1f874db2e69d8ac697ab41ff045b1d4873"} Sep 11 01:30:05.218011 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:05.217985 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bae8d3-29818170-v9hp5" Sep 11 01:30:06.092805 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:06.092733 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bae8d3-29818170-v9hp5" Sep 11 01:30:06.092805 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:06.092758 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bae8d3-29818170-v9hp5" event={"ID":"f20ce962-77ef-4451-bf8c-fcfaa2995c01","Type":"ContainerDied","Data":"32410a5677716997152195b9d258722207865b1848ce5e24177d817f910cb3c2"} Sep 11 01:30:06.093031 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:06.092819 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32410a5677716997152195b9d258722207865b1848ce5e24177d817f910cb3c2" Sep 11 01:30:13.830787 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:13.830750 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:30:18.581663 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:18.581628 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" podUID="ff83015b-8ad2-42da-9725-43905db25ee0" containerName="main" containerID="cri-o://e6d7e13de9bedefb7314948fd4d7cde8b28ed15255ceb3cdcba6a2f203c96fc8" gracePeriod=45 Sep 11 01:30:18.834401 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:18.834348 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:30:18.925673 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:18.925641 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68vh4\" (UniqueName: \"kubernetes.io/projected/ff83015b-8ad2-42da-9725-43905db25ee0-kube-api-access-68vh4\") pod \"ff83015b-8ad2-42da-9725-43905db25ee0\" (UID: \"ff83015b-8ad2-42da-9725-43905db25ee0\") " Sep 11 01:30:18.925823 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:18.925683 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff83015b-8ad2-42da-9725-43905db25ee0-model-cache\") pod \"ff83015b-8ad2-42da-9725-43905db25ee0\" (UID: \"ff83015b-8ad2-42da-9725-43905db25ee0\") " Sep 11 01:30:18.925823 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:18.925716 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff83015b-8ad2-42da-9725-43905db25ee0-tmp-dir\") pod \"ff83015b-8ad2-42da-9725-43905db25ee0\" (UID: \"ff83015b-8ad2-42da-9725-43905db25ee0\") " Sep 11 01:30:18.925823 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:18.925758 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff83015b-8ad2-42da-9725-43905db25ee0-home\") pod \"ff83015b-8ad2-42da-9725-43905db25ee0\" (UID: \"ff83015b-8ad2-42da-9725-43905db25ee0\") " Sep 11 01:30:18.925823 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:18.925780 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff83015b-8ad2-42da-9725-43905db25ee0-dshm\") pod \"ff83015b-8ad2-42da-9725-43905db25ee0\" (UID: \"ff83015b-8ad2-42da-9725-43905db25ee0\") " Sep 11 01:30:18.925823 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:18.925819 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff83015b-8ad2-42da-9725-43905db25ee0-tls-certs\") pod \"ff83015b-8ad2-42da-9725-43905db25ee0\" (UID: \"ff83015b-8ad2-42da-9725-43905db25ee0\") " Sep 11 01:30:18.926119 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:18.926038 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff83015b-8ad2-42da-9725-43905db25ee0-model-cache" (OuterVolumeSpecName: "model-cache") pod "ff83015b-8ad2-42da-9725-43905db25ee0" (UID: "ff83015b-8ad2-42da-9725-43905db25ee0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:30:18.926119 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:18.926051 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff83015b-8ad2-42da-9725-43905db25ee0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ff83015b-8ad2-42da-9725-43905db25ee0" (UID: "ff83015b-8ad2-42da-9725-43905db25ee0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:30:18.926119 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:18.926066 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff83015b-8ad2-42da-9725-43905db25ee0-home" (OuterVolumeSpecName: "home") pod "ff83015b-8ad2-42da-9725-43905db25ee0" (UID: "ff83015b-8ad2-42da-9725-43905db25ee0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:30:18.926275 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:18.926182 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff83015b-8ad2-42da-9725-43905db25ee0-model-cache\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:30:18.926275 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:18.926195 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff83015b-8ad2-42da-9725-43905db25ee0-tmp-dir\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:30:18.926275 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:18.926203 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff83015b-8ad2-42da-9725-43905db25ee0-home\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:30:18.927905 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:18.927875 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff83015b-8ad2-42da-9725-43905db25ee0-kube-api-access-68vh4" (OuterVolumeSpecName: "kube-api-access-68vh4") pod "ff83015b-8ad2-42da-9725-43905db25ee0" (UID: "ff83015b-8ad2-42da-9725-43905db25ee0"). InnerVolumeSpecName "kube-api-access-68vh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:30:18.928011 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:18.927912 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff83015b-8ad2-42da-9725-43905db25ee0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff83015b-8ad2-42da-9725-43905db25ee0" (UID: "ff83015b-8ad2-42da-9725-43905db25ee0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:30:18.928011 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:18.927942 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff83015b-8ad2-42da-9725-43905db25ee0-dshm" (OuterVolumeSpecName: "dshm") pod "ff83015b-8ad2-42da-9725-43905db25ee0" (UID: "ff83015b-8ad2-42da-9725-43905db25ee0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:30:19.026985 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:19.026954 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68vh4\" (UniqueName: \"kubernetes.io/projected/ff83015b-8ad2-42da-9725-43905db25ee0-kube-api-access-68vh4\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:30:19.027144 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:19.027128 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff83015b-8ad2-42da-9725-43905db25ee0-dshm\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:30:19.027205 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:19.027171 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff83015b-8ad2-42da-9725-43905db25ee0-tls-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:30:19.157789 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:19.157722 2554 generic.go:356] "Generic (PLEG): container finished" podID="ff83015b-8ad2-42da-9725-43905db25ee0" containerID="e6d7e13de9bedefb7314948fd4d7cde8b28ed15255ceb3cdcba6a2f203c96fc8" exitCode=0 Sep 11 01:30:19.157789 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:19.157770 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" Sep 11 01:30:19.157789 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:19.157773 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" event={"ID":"ff83015b-8ad2-42da-9725-43905db25ee0","Type":"ContainerDied","Data":"e6d7e13de9bedefb7314948fd4d7cde8b28ed15255ceb3cdcba6a2f203c96fc8"} Sep 11 01:30:19.157998 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:19.157815 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9" event={"ID":"ff83015b-8ad2-42da-9725-43905db25ee0","Type":"ContainerDied","Data":"5aca46501a0c1ce2edbf11b34f7133a9cfa11f68668ed78203c04ab1b351451c"} Sep 11 01:30:19.157998 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:19.157835 2554 scope.go:117] "RemoveContainer" containerID="e6d7e13de9bedefb7314948fd4d7cde8b28ed15255ceb3cdcba6a2f203c96fc8" Sep 11 01:30:19.167030 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:19.167015 2554 scope.go:117] "RemoveContainer" containerID="e6d7e13de9bedefb7314948fd4d7cde8b28ed15255ceb3cdcba6a2f203c96fc8" Sep 11 01:30:19.167273 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:30:19.167254 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6d7e13de9bedefb7314948fd4d7cde8b28ed15255ceb3cdcba6a2f203c96fc8\": container with ID starting with e6d7e13de9bedefb7314948fd4d7cde8b28ed15255ceb3cdcba6a2f203c96fc8 not found: ID does not exist" containerID="e6d7e13de9bedefb7314948fd4d7cde8b28ed15255ceb3cdcba6a2f203c96fc8" Sep 11 01:30:19.167339 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:19.167279 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d7e13de9bedefb7314948fd4d7cde8b28ed15255ceb3cdcba6a2f203c96fc8"} err="failed to get container status \"e6d7e13de9bedefb7314948fd4d7cde8b28ed15255ceb3cdcba6a2f203c96fc8\": rpc error: code = NotFound desc = could not find container \"e6d7e13de9bedefb7314948fd4d7cde8b28ed15255ceb3cdcba6a2f203c96fc8\": container with ID starting with e6d7e13de9bedefb7314948fd4d7cde8b28ed15255ceb3cdcba6a2f203c96fc8 not found: ID does not exist" Sep 11 01:30:19.178698 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:19.178675 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9"] Sep 11 01:30:19.182036 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:19.182018 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-976998/ipp-route-976998-kserve-67d77cb779-qdmb9"] Sep 11 01:30:19.678130 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:19.678102 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff83015b-8ad2-42da-9725-43905db25ee0" path="/var/lib/kubelet/pods/ff83015b-8ad2-42da-9725-43905db25ee0/volumes" Sep 11 01:30:40.333789 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:40.333752 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:30:40.759917 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:40.759872 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:30:46.309135 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:46.309056 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:30:46.368361 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:46.368329 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:30:46.598021 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:46.597939 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:30:47.150136 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:47.150101 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:30:57.329807 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:57.329770 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp"] Sep 11 01:30:57.330599 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:57.330562 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp" podUID="e1c56ab8-142c-45f8-950f-19c8f5087473" containerName="maas-api" containerID="cri-o://9a9bafe0c92285c9330d74940c5a2b56b8ff1894e1575527474a25d841992d1d" gracePeriod=30 Sep 11 01:30:57.361772 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:57.361727 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:30:57.362104 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:57.361993 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bae8d3-29818170-v9hp5"] Sep 11 01:30:57.363716 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:57.363692 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bae8d3-29818170-v9hp5"] Sep 11 01:30:57.423775 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:57.423740 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:30:57.479663 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:57.479625 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:30:57.576651 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:57.576609 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:30:57.585202 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:57.585157 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp" Sep 11 01:30:57.625422 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:57.625393 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cqj7\" (UniqueName: \"kubernetes.io/projected/e1c56ab8-142c-45f8-950f-19c8f5087473-kube-api-access-6cqj7\") pod \"e1c56ab8-142c-45f8-950f-19c8f5087473\" (UID: \"e1c56ab8-142c-45f8-950f-19c8f5087473\") " Sep 11 01:30:57.625528 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:57.625455 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1c56ab8-142c-45f8-950f-19c8f5087473-metrics-certs\") pod \"e1c56ab8-142c-45f8-950f-19c8f5087473\" (UID: \"e1c56ab8-142c-45f8-950f-19c8f5087473\") " Sep 11 01:30:57.625578 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:57.625543 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1c56ab8-142c-45f8-950f-19c8f5087473-maas-api-tls\") pod \"e1c56ab8-142c-45f8-950f-19c8f5087473\" (UID: \"e1c56ab8-142c-45f8-950f-19c8f5087473\") " Sep 11 01:30:57.625638 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:57.625580 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1c56ab8-142c-45f8-950f-19c8f5087473-openshift-service-ca\") pod \"e1c56ab8-142c-45f8-950f-19c8f5087473\" (UID: \"e1c56ab8-142c-45f8-950f-19c8f5087473\") " Sep 11 01:30:57.626030 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:57.625991 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1c56ab8-142c-45f8-950f-19c8f5087473-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e1c56ab8-142c-45f8-950f-19c8f5087473" (UID: "e1c56ab8-142c-45f8-950f-19c8f5087473"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:30:57.627563 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:57.627529 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1c56ab8-142c-45f8-950f-19c8f5087473-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e1c56ab8-142c-45f8-950f-19c8f5087473" (UID: "e1c56ab8-142c-45f8-950f-19c8f5087473"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:30:57.627655 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:57.627589 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1c56ab8-142c-45f8-950f-19c8f5087473-kube-api-access-6cqj7" (OuterVolumeSpecName: "kube-api-access-6cqj7") pod "e1c56ab8-142c-45f8-950f-19c8f5087473" (UID: "e1c56ab8-142c-45f8-950f-19c8f5087473"). InnerVolumeSpecName "kube-api-access-6cqj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:30:57.627655 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:57.627624 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1c56ab8-142c-45f8-950f-19c8f5087473-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e1c56ab8-142c-45f8-950f-19c8f5087473" (UID: "e1c56ab8-142c-45f8-950f-19c8f5087473"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:30:57.668187 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:57.668161 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:30:57.677741 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:57.677718 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f20ce962-77ef-4451-bf8c-fcfaa2995c01" path="/var/lib/kubelet/pods/f20ce962-77ef-4451-bf8c-fcfaa2995c01/volumes" Sep 11 01:30:57.726180 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:57.726154 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cqj7\" (UniqueName: \"kubernetes.io/projected/e1c56ab8-142c-45f8-950f-19c8f5087473-kube-api-access-6cqj7\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:30:57.726180 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:57.726176 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1c56ab8-142c-45f8-950f-19c8f5087473-metrics-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:30:57.726309 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:57.726190 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1c56ab8-142c-45f8-950f-19c8f5087473-maas-api-tls\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:30:57.726309 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:57.726202 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1c56ab8-142c-45f8-950f-19c8f5087473-openshift-service-ca\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:30:58.001866 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:58.001815 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:30:58.351263 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:58.351186 2554 generic.go:356] "Generic (PLEG): container finished" podID="e1c56ab8-142c-45f8-950f-19c8f5087473" containerID="9a9bafe0c92285c9330d74940c5a2b56b8ff1894e1575527474a25d841992d1d" exitCode=0 Sep 11 01:30:58.351263 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:58.351240 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp" Sep 11 01:30:58.351697 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:58.351277 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp" event={"ID":"e1c56ab8-142c-45f8-950f-19c8f5087473","Type":"ContainerDied","Data":"9a9bafe0c92285c9330d74940c5a2b56b8ff1894e1575527474a25d841992d1d"} Sep 11 01:30:58.351697 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:58.351352 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp" event={"ID":"e1c56ab8-142c-45f8-950f-19c8f5087473","Type":"ContainerDied","Data":"06aaf7fb81d2ede99bf05ed63b2a60aff0d965a3bb3ed538ccef141acf38583f"} Sep 11 01:30:58.351697 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:58.351384 2554 scope.go:117] "RemoveContainer" containerID="9a9bafe0c92285c9330d74940c5a2b56b8ff1894e1575527474a25d841992d1d" Sep 11 01:30:58.360645 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:58.360630 2554 scope.go:117] "RemoveContainer" containerID="9a9bafe0c92285c9330d74940c5a2b56b8ff1894e1575527474a25d841992d1d" Sep 11 01:30:58.360952 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:30:58.360929 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a9bafe0c92285c9330d74940c5a2b56b8ff1894e1575527474a25d841992d1d\": container with ID starting with 9a9bafe0c92285c9330d74940c5a2b56b8ff1894e1575527474a25d841992d1d not found: ID does not exist" containerID="9a9bafe0c92285c9330d74940c5a2b56b8ff1894e1575527474a25d841992d1d" Sep 11 01:30:58.361053 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:58.360958 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a9bafe0c92285c9330d74940c5a2b56b8ff1894e1575527474a25d841992d1d"} err="failed to get container status \"9a9bafe0c92285c9330d74940c5a2b56b8ff1894e1575527474a25d841992d1d\": rpc error: code = NotFound desc = could not find container \"9a9bafe0c92285c9330d74940c5a2b56b8ff1894e1575527474a25d841992d1d\": container with ID starting with 9a9bafe0c92285c9330d74940c5a2b56b8ff1894e1575527474a25d841992d1d not found: ID does not exist" Sep 11 01:30:58.368461 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:58.368439 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp"] Sep 11 01:30:58.374261 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:58.374239 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bae8d3-7c7dbb54f5-cxmtp"] Sep 11 01:30:59.680658 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:30:59.680624 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1c56ab8-142c-45f8-950f-19c8f5087473" path="/var/lib/kubelet/pods/e1c56ab8-142c-45f8-950f-19c8f5087473/volumes" Sep 11 01:31:03.810960 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:03.810927 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:31:03.876044 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:03.876022 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:31:04.094374 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:04.094301 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:31:04.547803 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:04.547767 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:31:15.035336 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:15.035291 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:31:15.101227 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:15.101190 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:31:15.200348 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:15.200314 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:31:15.275649 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:15.275615 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:31:15.608278 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:15.608243 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:31:21.466645 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:21.466609 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:31:21.567008 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:21.566972 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:31:21.741867 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:21.741762 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:31:22.219495 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:22.219465 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:31:32.521162 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:32.521132 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:31:32.597484 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:32.597451 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:31:32.659913 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:32.659826 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:31:32.718970 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:32.718941 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:31:33.086434 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:33.086400 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:31:38.969140 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:38.969107 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:31:39.067112 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:39.067077 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:31:39.245915 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:39.245823 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:31:41.887130 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:41.887091 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8f8d984c8-dbdv8"] Sep 11 01:31:41.887614 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:41.887318 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-8f8d984c8-dbdv8" podUID="b40367b4-cbb4-4bed-ab22-27752879bbd0" containerName="manager" containerID="cri-o://b10eef12c723b85220f52c5034b984e139195b6edeaf7566ea82bad9ecd57c26" gracePeriod=30 Sep 11 01:31:42.137061 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:42.137036 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8f8d984c8-dbdv8" Sep 11 01:31:42.295499 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:42.295473 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b40367b4-cbb4-4bed-ab22-27752879bbd0-webhook-cert\") pod \"b40367b4-cbb4-4bed-ab22-27752879bbd0\" (UID: \"b40367b4-cbb4-4bed-ab22-27752879bbd0\") " Sep 11 01:31:42.295665 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:42.295513 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b40367b4-cbb4-4bed-ab22-27752879bbd0-metrics-certs\") pod \"b40367b4-cbb4-4bed-ab22-27752879bbd0\" (UID: \"b40367b4-cbb4-4bed-ab22-27752879bbd0\") " Sep 11 01:31:42.295665 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:42.295593 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68jgw\" (UniqueName: \"kubernetes.io/projected/b40367b4-cbb4-4bed-ab22-27752879bbd0-kube-api-access-68jgw\") pod \"b40367b4-cbb4-4bed-ab22-27752879bbd0\" (UID: \"b40367b4-cbb4-4bed-ab22-27752879bbd0\") " Sep 11 01:31:42.297694 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:42.297666 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b40367b4-cbb4-4bed-ab22-27752879bbd0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b40367b4-cbb4-4bed-ab22-27752879bbd0" (UID: "b40367b4-cbb4-4bed-ab22-27752879bbd0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:31:42.297836 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:42.297743 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b40367b4-cbb4-4bed-ab22-27752879bbd0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b40367b4-cbb4-4bed-ab22-27752879bbd0" (UID: "b40367b4-cbb4-4bed-ab22-27752879bbd0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:31:42.297836 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:42.297790 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40367b4-cbb4-4bed-ab22-27752879bbd0-kube-api-access-68jgw" (OuterVolumeSpecName: "kube-api-access-68jgw") pod "b40367b4-cbb4-4bed-ab22-27752879bbd0" (UID: "b40367b4-cbb4-4bed-ab22-27752879bbd0"). InnerVolumeSpecName "kube-api-access-68jgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:31:42.396820 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:42.396790 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68jgw\" (UniqueName: \"kubernetes.io/projected/b40367b4-cbb4-4bed-ab22-27752879bbd0-kube-api-access-68jgw\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:31:42.396820 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:42.396811 2554 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b40367b4-cbb4-4bed-ab22-27752879bbd0-webhook-cert\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:31:42.396820 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:42.396820 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b40367b4-cbb4-4bed-ab22-27752879bbd0-metrics-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:31:42.564671 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:42.564600 2554 generic.go:356] "Generic (PLEG): container finished" podID="b40367b4-cbb4-4bed-ab22-27752879bbd0" containerID="b10eef12c723b85220f52c5034b984e139195b6edeaf7566ea82bad9ecd57c26" exitCode=0 Sep 11 01:31:42.564671 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:42.564642 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8f8d984c8-dbdv8" Sep 11 01:31:42.564671 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:42.564664 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8f8d984c8-dbdv8" event={"ID":"b40367b4-cbb4-4bed-ab22-27752879bbd0","Type":"ContainerDied","Data":"b10eef12c723b85220f52c5034b984e139195b6edeaf7566ea82bad9ecd57c26"} Sep 11 01:31:42.564918 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:42.564707 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8f8d984c8-dbdv8" event={"ID":"b40367b4-cbb4-4bed-ab22-27752879bbd0","Type":"ContainerDied","Data":"dc8145064676f6bebc7620262427e34fa2ab5b6ae3d20274e48a5918b5851ec8"} Sep 11 01:31:42.564918 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:42.564728 2554 scope.go:117] "RemoveContainer" containerID="b10eef12c723b85220f52c5034b984e139195b6edeaf7566ea82bad9ecd57c26" Sep 11 01:31:42.574186 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:42.574169 2554 scope.go:117] "RemoveContainer" containerID="b10eef12c723b85220f52c5034b984e139195b6edeaf7566ea82bad9ecd57c26" Sep 11 01:31:42.574430 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:31:42.574414 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b10eef12c723b85220f52c5034b984e139195b6edeaf7566ea82bad9ecd57c26\": container with ID starting with b10eef12c723b85220f52c5034b984e139195b6edeaf7566ea82bad9ecd57c26 not found: ID does not exist" containerID="b10eef12c723b85220f52c5034b984e139195b6edeaf7566ea82bad9ecd57c26" Sep 11 01:31:42.574479 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:42.574436 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10eef12c723b85220f52c5034b984e139195b6edeaf7566ea82bad9ecd57c26"} err="failed to get container status \"b10eef12c723b85220f52c5034b984e139195b6edeaf7566ea82bad9ecd57c26\": rpc error: code = NotFound desc = could not find container \"b10eef12c723b85220f52c5034b984e139195b6edeaf7566ea82bad9ecd57c26\": container with ID starting with b10eef12c723b85220f52c5034b984e139195b6edeaf7566ea82bad9ecd57c26 not found: ID does not exist" Sep 11 01:31:42.586677 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:42.586651 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8f8d984c8-dbdv8"] Sep 11 01:31:42.588262 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:42.588243 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8f8d984c8-dbdv8"] Sep 11 01:31:43.677749 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:31:43.677714 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b40367b4-cbb4-4bed-ab22-27752879bbd0" path="/var/lib/kubelet/pods/b40367b4-cbb4-4bed-ab22-27752879bbd0/volumes" Sep 11 01:32:53.753972 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:32:53.753941 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wtd8v_c2b320dd-43e9-435d-8e4f-2bca1f1233ff/ovn-acl-logging/0.log" Sep 11 01:32:53.757825 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:32:53.757803 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wtd8v_c2b320dd-43e9-435d-8e4f-2bca1f1233ff/ovn-acl-logging/0.log" Sep 11 01:32:55.477593 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:32:55.477564 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:33:04.201437 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:33:04.201403 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:33:05.474078 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:33:05.474047 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:33:07.059036 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:33:07.059002 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:33:25.938999 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:33:25.938960 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:33:25.999342 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:33:25.999313 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:34:20.450821 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:34:20.450743 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kdjzs"] Sep 11 01:34:20.451345 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:34:20.450977 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kdjzs" podUID="b75f844c-c91a-485a-82db-420603cdbbae" containerName="manager" containerID="cri-o://f723ffa3196d38165a0eb0b51723a4f992ed430cb80b49ba9f73bf2a1ee5d2b2" gracePeriod=10 Sep 11 01:34:21.192635 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:34:21.192610 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kdjzs" Sep 11 01:34:21.221294 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:34:21.221262 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgx86\" (UniqueName: \"kubernetes.io/projected/b75f844c-c91a-485a-82db-420603cdbbae-kube-api-access-zgx86\") pod \"b75f844c-c91a-485a-82db-420603cdbbae\" (UID: \"b75f844c-c91a-485a-82db-420603cdbbae\") " Sep 11 01:34:21.221478 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:34:21.221361 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b75f844c-c91a-485a-82db-420603cdbbae-extensions-socket-volume\") pod \"b75f844c-c91a-485a-82db-420603cdbbae\" (UID: \"b75f844c-c91a-485a-82db-420603cdbbae\") " Sep 11 01:34:21.221758 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:34:21.221733 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75f844c-c91a-485a-82db-420603cdbbae-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b75f844c-c91a-485a-82db-420603cdbbae" (UID: "b75f844c-c91a-485a-82db-420603cdbbae"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 01:34:21.223418 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:34:21.223393 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75f844c-c91a-485a-82db-420603cdbbae-kube-api-access-zgx86" (OuterVolumeSpecName: "kube-api-access-zgx86") pod "b75f844c-c91a-485a-82db-420603cdbbae" (UID: "b75f844c-c91a-485a-82db-420603cdbbae"). InnerVolumeSpecName "kube-api-access-zgx86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:34:21.319086 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:34:21.319011 2554 generic.go:356] "Generic (PLEG): container finished" podID="b75f844c-c91a-485a-82db-420603cdbbae" containerID="f723ffa3196d38165a0eb0b51723a4f992ed430cb80b49ba9f73bf2a1ee5d2b2" exitCode=0 Sep 11 01:34:21.319086 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:34:21.319053 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kdjzs" Sep 11 01:34:21.319278 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:34:21.319093 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kdjzs" event={"ID":"b75f844c-c91a-485a-82db-420603cdbbae","Type":"ContainerDied","Data":"f723ffa3196d38165a0eb0b51723a4f992ed430cb80b49ba9f73bf2a1ee5d2b2"} Sep 11 01:34:21.319278 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:34:21.319151 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kdjzs" event={"ID":"b75f844c-c91a-485a-82db-420603cdbbae","Type":"ContainerDied","Data":"babbe9989442d35e564c2adeee1af51eb71869b9618e0ac99b066cef021145ea"} Sep 11 01:34:21.319278 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:34:21.319179 2554 scope.go:117] "RemoveContainer" containerID="f723ffa3196d38165a0eb0b51723a4f992ed430cb80b49ba9f73bf2a1ee5d2b2" Sep 11 01:34:21.322233 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:34:21.322211 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b75f844c-c91a-485a-82db-420603cdbbae-extensions-socket-volume\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:34:21.322233 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:34:21.322234 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgx86\" (UniqueName: \"kubernetes.io/projected/b75f844c-c91a-485a-82db-420603cdbbae-kube-api-access-zgx86\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:34:21.328107 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:34:21.328092 2554 scope.go:117] "RemoveContainer" containerID="f723ffa3196d38165a0eb0b51723a4f992ed430cb80b49ba9f73bf2a1ee5d2b2" Sep 11 01:34:21.328363 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:34:21.328348 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f723ffa3196d38165a0eb0b51723a4f992ed430cb80b49ba9f73bf2a1ee5d2b2\": container with ID starting with f723ffa3196d38165a0eb0b51723a4f992ed430cb80b49ba9f73bf2a1ee5d2b2 not found: ID does not exist" containerID="f723ffa3196d38165a0eb0b51723a4f992ed430cb80b49ba9f73bf2a1ee5d2b2" Sep 11 01:34:21.328410 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:34:21.328371 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f723ffa3196d38165a0eb0b51723a4f992ed430cb80b49ba9f73bf2a1ee5d2b2"} err="failed to get container status \"f723ffa3196d38165a0eb0b51723a4f992ed430cb80b49ba9f73bf2a1ee5d2b2\": rpc error: code = NotFound desc = could not find container \"f723ffa3196d38165a0eb0b51723a4f992ed430cb80b49ba9f73bf2a1ee5d2b2\": container with ID starting with f723ffa3196d38165a0eb0b51723a4f992ed430cb80b49ba9f73bf2a1ee5d2b2 not found: ID does not exist" Sep 11 01:34:21.340245 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:34:21.340221 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kdjzs"] Sep 11 01:34:21.344227 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:34:21.344204 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kdjzs"] Sep 11 01:34:21.678393 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:34:21.678340 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b75f844c-c91a-485a-82db-420603cdbbae" path="/var/lib/kubelet/pods/b75f844c-c91a-485a-82db-420603cdbbae/volumes" Sep 11 01:35:01.096467 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:01.096437 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:35:03.297247 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:03.297206 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:35:21.156044 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.156008 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sjfvq"] Sep 11 01:35:21.156417 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.156361 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f20ce962-77ef-4451-bf8c-fcfaa2995c01" containerName="cleanup" Sep 11 01:35:21.156417 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.156373 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f20ce962-77ef-4451-bf8c-fcfaa2995c01" containerName="cleanup" Sep 11 01:35:21.156417 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.156383 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1c56ab8-142c-45f8-950f-19c8f5087473" containerName="maas-api" Sep 11 01:35:21.156417 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.156388 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1c56ab8-142c-45f8-950f-19c8f5087473" containerName="maas-api" Sep 11 01:35:21.156417 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.156396 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff83015b-8ad2-42da-9725-43905db25ee0" containerName="main" Sep 11 01:35:21.156417 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.156401 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff83015b-8ad2-42da-9725-43905db25ee0" containerName="main" Sep 11 01:35:21.156417 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.156409 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b75f844c-c91a-485a-82db-420603cdbbae" containerName="manager" Sep 11 01:35:21.156417 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.156414 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b75f844c-c91a-485a-82db-420603cdbbae" containerName="manager" Sep 11 01:35:21.156417 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.156421 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b40367b4-cbb4-4bed-ab22-27752879bbd0" containerName="manager" Sep 11 01:35:21.156702 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.156426 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b40367b4-cbb4-4bed-ab22-27752879bbd0" containerName="manager" Sep 11 01:35:21.156702 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.156485 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b75f844c-c91a-485a-82db-420603cdbbae" containerName="manager" Sep 11 01:35:21.156702 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.156494 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f20ce962-77ef-4451-bf8c-fcfaa2995c01" containerName="cleanup" Sep 11 01:35:21.156702 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.156501 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1c56ab8-142c-45f8-950f-19c8f5087473" containerName="maas-api" Sep 11 01:35:21.156702 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.156507 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff83015b-8ad2-42da-9725-43905db25ee0" containerName="main" Sep 11 01:35:21.156702 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.156513 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b40367b4-cbb4-4bed-ab22-27752879bbd0" containerName="manager" Sep 11 01:35:21.159462 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.159445 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sjfvq" Sep 11 01:35:21.162539 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.162515 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sstc2\"" Sep 11 01:35:21.170505 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.170483 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sjfvq"] Sep 11 01:35:21.180110 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.180088 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a63eccf-904b-458e-b080-355b92e5bc19-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-sjfvq\" (UID: \"1a63eccf-904b-458e-b080-355b92e5bc19\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sjfvq" Sep 11 01:35:21.180226 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.180131 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrk6h\" (UniqueName: \"kubernetes.io/projected/1a63eccf-904b-458e-b080-355b92e5bc19-kube-api-access-mrk6h\") pod \"kuadrant-operator-controller-manager-5769f99d9f-sjfvq\" (UID: \"1a63eccf-904b-458e-b080-355b92e5bc19\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sjfvq" Sep 11 01:35:21.280599 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.280566 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a63eccf-904b-458e-b080-355b92e5bc19-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-sjfvq\" (UID: \"1a63eccf-904b-458e-b080-355b92e5bc19\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sjfvq" Sep 11 01:35:21.280759 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.280621 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrk6h\" (UniqueName: \"kubernetes.io/projected/1a63eccf-904b-458e-b080-355b92e5bc19-kube-api-access-mrk6h\") pod \"kuadrant-operator-controller-manager-5769f99d9f-sjfvq\" (UID: \"1a63eccf-904b-458e-b080-355b92e5bc19\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sjfvq" Sep 11 01:35:21.280952 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.280934 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a63eccf-904b-458e-b080-355b92e5bc19-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-sjfvq\" (UID: \"1a63eccf-904b-458e-b080-355b92e5bc19\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sjfvq" Sep 11 01:35:21.288950 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.288924 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrk6h\" (UniqueName: \"kubernetes.io/projected/1a63eccf-904b-458e-b080-355b92e5bc19-kube-api-access-mrk6h\") pod \"kuadrant-operator-controller-manager-5769f99d9f-sjfvq\" (UID: \"1a63eccf-904b-458e-b080-355b92e5bc19\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sjfvq" Sep 11 01:35:21.469438 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.469407 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sjfvq" Sep 11 01:35:21.744196 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:21.744171 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sjfvq"] Sep 11 01:35:21.747137 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:35:21.747106 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a63eccf_904b_458e_b080_355b92e5bc19.slice/crio-e72773bc62764a87f3bf591726de1baa640e36bed8fc456ff8861854fc29bb9f WatchSource:0}: Error finding container e72773bc62764a87f3bf591726de1baa640e36bed8fc456ff8861854fc29bb9f: Status 404 returned error can't find the container with id e72773bc62764a87f3bf591726de1baa640e36bed8fc456ff8861854fc29bb9f Sep 11 01:35:22.606421 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:22.606379 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sjfvq" event={"ID":"1a63eccf-904b-458e-b080-355b92e5bc19","Type":"ContainerStarted","Data":"5cec4d064a12886761d5f17b2c120846f115f1295939405295b17747b8ef7b42"} Sep 11 01:35:22.606785 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:22.606431 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sjfvq" event={"ID":"1a63eccf-904b-458e-b080-355b92e5bc19","Type":"ContainerStarted","Data":"e72773bc62764a87f3bf591726de1baa640e36bed8fc456ff8861854fc29bb9f"} Sep 11 01:35:22.606785 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:22.606494 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sjfvq" Sep 11 01:35:22.631070 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:22.631027 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sjfvq" podStartSLOduration=1.6310143689999999 podStartE2EDuration="1.631014369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:35:22.630022444 +0000 UTC m=+2249.453475991" watchObservedRunningTime="2026-09-11 01:35:22.631014369 +0000 UTC m=+2249.454467879" Sep 11 01:35:33.613587 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:35:33.613553 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sjfvq" Sep 11 01:36:08.890623 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:36:08.890582 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:36:09.788949 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:36:09.788902 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:36:10.072422 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:36:10.072335 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:36:16.018334 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:36:16.018300 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:36:42.429364 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:36:42.429328 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:36:43.139683 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:36:43.139651 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:36:54.022719 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:36:54.022637 2554 scope.go:117] "RemoveContainer" containerID="33673718c25651313269cb4ba2218c1f874db2e69d8ac697ab41ff045b1d4873" Sep 11 01:37:07.714027 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:07.713996 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-fcdb7dc4c-7hp6h"] Sep 11 01:37:07.717321 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:07.717303 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-7hp6h" Sep 11 01:37:07.727126 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:07.727103 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-fcdb7dc4c-7hp6h"] Sep 11 01:37:07.802907 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:07.802880 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21c79eaa-77a7-40e8-a1fe-7aff22923fde-maas-api-tls\") pod \"maas-api-fcdb7dc4c-7hp6h\" (UID: \"21c79eaa-77a7-40e8-a1fe-7aff22923fde\") " pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-7hp6h" Sep 11 01:37:07.803032 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:07.802935 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21c79eaa-77a7-40e8-a1fe-7aff22923fde-metrics-certs\") pod \"maas-api-fcdb7dc4c-7hp6h\" (UID: \"21c79eaa-77a7-40e8-a1fe-7aff22923fde\") " pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-7hp6h" Sep 11 01:37:07.803097 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:07.803080 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-925qp\" (UniqueName: \"kubernetes.io/projected/21c79eaa-77a7-40e8-a1fe-7aff22923fde-kube-api-access-925qp\") pod \"maas-api-fcdb7dc4c-7hp6h\" (UID: \"21c79eaa-77a7-40e8-a1fe-7aff22923fde\") " pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-7hp6h" Sep 11 01:37:07.803140 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:07.803118 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21c79eaa-77a7-40e8-a1fe-7aff22923fde-openshift-service-ca\") pod \"maas-api-fcdb7dc4c-7hp6h\" (UID: \"21c79eaa-77a7-40e8-a1fe-7aff22923fde\") " pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-7hp6h" Sep 11 01:37:07.903568 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:07.903542 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21c79eaa-77a7-40e8-a1fe-7aff22923fde-maas-api-tls\") pod \"maas-api-fcdb7dc4c-7hp6h\" (UID: \"21c79eaa-77a7-40e8-a1fe-7aff22923fde\") " pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-7hp6h" Sep 11 01:37:07.903706 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:07.903581 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21c79eaa-77a7-40e8-a1fe-7aff22923fde-metrics-certs\") pod \"maas-api-fcdb7dc4c-7hp6h\" (UID: \"21c79eaa-77a7-40e8-a1fe-7aff22923fde\") " pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-7hp6h" Sep 11 01:37:07.903706 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:07.903617 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-925qp\" (UniqueName: \"kubernetes.io/projected/21c79eaa-77a7-40e8-a1fe-7aff22923fde-kube-api-access-925qp\") pod \"maas-api-fcdb7dc4c-7hp6h\" (UID: \"21c79eaa-77a7-40e8-a1fe-7aff22923fde\") " pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-7hp6h" Sep 11 01:37:07.903706 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:07.903644 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21c79eaa-77a7-40e8-a1fe-7aff22923fde-openshift-service-ca\") pod \"maas-api-fcdb7dc4c-7hp6h\" (UID: \"21c79eaa-77a7-40e8-a1fe-7aff22923fde\") " pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-7hp6h" Sep 11 01:37:07.904312 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:07.904290 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21c79eaa-77a7-40e8-a1fe-7aff22923fde-openshift-service-ca\") pod \"maas-api-fcdb7dc4c-7hp6h\" (UID: \"21c79eaa-77a7-40e8-a1fe-7aff22923fde\") " pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-7hp6h" Sep 11 01:37:07.906014 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:07.905985 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21c79eaa-77a7-40e8-a1fe-7aff22923fde-maas-api-tls\") pod \"maas-api-fcdb7dc4c-7hp6h\" (UID: \"21c79eaa-77a7-40e8-a1fe-7aff22923fde\") " pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-7hp6h" Sep 11 01:37:07.906159 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:07.906139 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21c79eaa-77a7-40e8-a1fe-7aff22923fde-metrics-certs\") pod \"maas-api-fcdb7dc4c-7hp6h\" (UID: \"21c79eaa-77a7-40e8-a1fe-7aff22923fde\") " pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-7hp6h" Sep 11 01:37:07.910979 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:07.910962 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-925qp\" (UniqueName: \"kubernetes.io/projected/21c79eaa-77a7-40e8-a1fe-7aff22923fde-kube-api-access-925qp\") pod \"maas-api-fcdb7dc4c-7hp6h\" (UID: \"21c79eaa-77a7-40e8-a1fe-7aff22923fde\") " pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-7hp6h" Sep 11 01:37:08.029584 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:08.029519 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-7hp6h" Sep 11 01:37:08.299993 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:08.299970 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-fcdb7dc4c-7hp6h"] Sep 11 01:37:08.302342 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:37:08.302314 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c79eaa_77a7_40e8_a1fe_7aff22923fde.slice/crio-4db6222c2293ca3e68b1eafdf756492a4eab0bb03adb3aa77c1aae408f0091bc WatchSource:0}: Error finding container 4db6222c2293ca3e68b1eafdf756492a4eab0bb03adb3aa77c1aae408f0091bc: Status 404 returned error can't find the container with id 4db6222c2293ca3e68b1eafdf756492a4eab0bb03adb3aa77c1aae408f0091bc Sep 11 01:37:08.303677 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:08.303657 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 01:37:09.101418 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:09.101386 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-7hp6h" event={"ID":"21c79eaa-77a7-40e8-a1fe-7aff22923fde","Type":"ContainerStarted","Data":"4f763a13e11a9a4cd9e1da25a9f247b0d6f7ec97631e1d18b4b70d94ac738937"} Sep 11 01:37:09.101780 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:09.101434 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-7hp6h" event={"ID":"21c79eaa-77a7-40e8-a1fe-7aff22923fde","Type":"ContainerStarted","Data":"4db6222c2293ca3e68b1eafdf756492a4eab0bb03adb3aa77c1aae408f0091bc"} Sep 11 01:37:09.101780 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:09.101480 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-7hp6h" Sep 11 01:37:09.118006 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:09.117965 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-7hp6h" podStartSLOduration=1.783264275 podStartE2EDuration="2.117950291s" totalImagesPullingTime="334.686016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 01:37:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:37:09.115623748 +0000 UTC m=+2355.939077259" watchObservedRunningTime="2026-09-11 01:37:09.117950291 +0000 UTC m=+2355.941403801" Sep 11 01:37:15.111692 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:15.111656 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-fcdb7dc4c-7hp6h" Sep 11 01:37:15.155017 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:15.154980 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5"] Sep 11 01:37:15.155192 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:15.155169 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5" podUID="4ba4726e-2615-4686-9fb9-f490c7886591" containerName="maas-api" containerID="cri-o://b5e39a40e5ecc16bc2404eca491216ca9046a2785edb9a3196231a1df1e1112c" gracePeriod=30 Sep 11 01:37:16.394337 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:16.394312 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5" Sep 11 01:37:16.472119 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:16.472091 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba4726e-2615-4686-9fb9-f490c7886591-openshift-service-ca\") pod \"4ba4726e-2615-4686-9fb9-f490c7886591\" (UID: \"4ba4726e-2615-4686-9fb9-f490c7886591\") " Sep 11 01:37:16.472248 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:16.472123 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vb68\" (UniqueName: \"kubernetes.io/projected/4ba4726e-2615-4686-9fb9-f490c7886591-kube-api-access-6vb68\") pod \"4ba4726e-2615-4686-9fb9-f490c7886591\" (UID: \"4ba4726e-2615-4686-9fb9-f490c7886591\") " Sep 11 01:37:16.472248 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:16.472218 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ba4726e-2615-4686-9fb9-f490c7886591-maas-api-tls\") pod \"4ba4726e-2615-4686-9fb9-f490c7886591\" (UID: \"4ba4726e-2615-4686-9fb9-f490c7886591\") " Sep 11 01:37:16.472248 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:16.472244 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ba4726e-2615-4686-9fb9-f490c7886591-metrics-certs\") pod \"4ba4726e-2615-4686-9fb9-f490c7886591\" (UID: \"4ba4726e-2615-4686-9fb9-f490c7886591\") " Sep 11 01:37:16.472459 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:16.472437 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba4726e-2615-4686-9fb9-f490c7886591-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4ba4726e-2615-4686-9fb9-f490c7886591" (UID: "4ba4726e-2615-4686-9fb9-f490c7886591"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 01:37:16.472534 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:16.472505 2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba4726e-2615-4686-9fb9-f490c7886591-openshift-service-ca\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:37:16.474312 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:16.474258 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba4726e-2615-4686-9fb9-f490c7886591-kube-api-access-6vb68" (OuterVolumeSpecName: "kube-api-access-6vb68") pod "4ba4726e-2615-4686-9fb9-f490c7886591" (UID: "4ba4726e-2615-4686-9fb9-f490c7886591"). InnerVolumeSpecName "kube-api-access-6vb68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 01:37:16.474312 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:16.474258 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ba4726e-2615-4686-9fb9-f490c7886591-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4ba4726e-2615-4686-9fb9-f490c7886591" (UID: "4ba4726e-2615-4686-9fb9-f490c7886591"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:37:16.474312 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:16.474300 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ba4726e-2615-4686-9fb9-f490c7886591-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4ba4726e-2615-4686-9fb9-f490c7886591" (UID: "4ba4726e-2615-4686-9fb9-f490c7886591"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 01:37:16.573778 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:16.573755 2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ba4726e-2615-4686-9fb9-f490c7886591-maas-api-tls\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:37:16.573778 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:16.573775 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ba4726e-2615-4686-9fb9-f490c7886591-metrics-certs\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:37:16.573933 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:16.573784 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vb68\" (UniqueName: \"kubernetes.io/projected/4ba4726e-2615-4686-9fb9-f490c7886591-kube-api-access-6vb68\") on node \"ip-10-0-135-107.ec2.internal\" DevicePath \"\"" Sep 11 01:37:17.139548 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:17.139517 2554 generic.go:356] "Generic (PLEG): container finished" podID="4ba4726e-2615-4686-9fb9-f490c7886591" containerID="b5e39a40e5ecc16bc2404eca491216ca9046a2785edb9a3196231a1df1e1112c" exitCode=0 Sep 11 01:37:17.139710 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:17.139556 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5" Sep 11 01:37:17.139710 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:17.139591 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5" event={"ID":"4ba4726e-2615-4686-9fb9-f490c7886591","Type":"ContainerDied","Data":"b5e39a40e5ecc16bc2404eca491216ca9046a2785edb9a3196231a1df1e1112c"} Sep 11 01:37:17.139710 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:17.139641 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5" event={"ID":"4ba4726e-2615-4686-9fb9-f490c7886591","Type":"ContainerDied","Data":"fa9c884d1331af30a1272d1b03f45791d241f0e1298df29614a9e595442443b8"} Sep 11 01:37:17.139710 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:17.139664 2554 scope.go:117] "RemoveContainer" containerID="b5e39a40e5ecc16bc2404eca491216ca9046a2785edb9a3196231a1df1e1112c" Sep 11 01:37:17.147873 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:17.147834 2554 scope.go:117] "RemoveContainer" containerID="b5e39a40e5ecc16bc2404eca491216ca9046a2785edb9a3196231a1df1e1112c" Sep 11 01:37:17.148124 ip-10-0-135-107 kubenswrapper[2554]: E0911 01:37:17.148108 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5e39a40e5ecc16bc2404eca491216ca9046a2785edb9a3196231a1df1e1112c\": container with ID starting with b5e39a40e5ecc16bc2404eca491216ca9046a2785edb9a3196231a1df1e1112c not found: ID does not exist" containerID="b5e39a40e5ecc16bc2404eca491216ca9046a2785edb9a3196231a1df1e1112c" Sep 11 01:37:17.148172 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:17.148133 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e39a40e5ecc16bc2404eca491216ca9046a2785edb9a3196231a1df1e1112c"} err="failed to get container status \"b5e39a40e5ecc16bc2404eca491216ca9046a2785edb9a3196231a1df1e1112c\": rpc error: code = NotFound desc = could not find container \"b5e39a40e5ecc16bc2404eca491216ca9046a2785edb9a3196231a1df1e1112c\": container with ID starting with b5e39a40e5ecc16bc2404eca491216ca9046a2785edb9a3196231a1df1e1112c not found: ID does not exist" Sep 11 01:37:17.162065 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:17.162041 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5"] Sep 11 01:37:17.166294 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:17.166272 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b7ff8bc87-b6ls5"] Sep 11 01:37:17.678106 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:17.678076 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ba4726e-2615-4686-9fb9-f490c7886591" path="/var/lib/kubelet/pods/4ba4726e-2615-4686-9fb9-f490c7886591/volumes" Sep 11 01:37:24.383784 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:24.383749 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:37:30.045028 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:30.045000 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78cd557cc4-ptpp5_39c32d6f-b4ab-4f08-b9d8-2b5fd0998ba5/authorino/0.log" Sep 11 01:37:35.392449 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:35.392408 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-kfzfx_7ddea17b-8323-4ad1-8171-a6e6e00f3133/manager/0.log" Sep 11 01:37:35.509446 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:35.509413 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-4ktqt_0cf5f7e0-6bb4-4349-9806-42540500a8d2/manager/0.log" Sep 11 01:37:35.756383 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:35.756360 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-6mqjx_412594fe-d273-4015-87ae-48e593f0f257/server/0.log" Sep 11 01:37:36.262304 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:36.262281 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c49b45946-qpjtx_5115bd54-60ec-4cd8-9e8e-06436e61adf0/copy-manifests/0.log" Sep 11 01:37:36.268060 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:36.268038 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c49b45946-qpjtx_5115bd54-60ec-4cd8-9e8e-06436e61adf0/manager/0.log" Sep 11 01:37:36.598479 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:36.598417 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-fcdb7dc4c-7hp6h_21c79eaa-77a7-40e8-a1fe-7aff22923fde/maas-api/0.log" Sep 11 01:37:37.051058 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:37.051039 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-djjsb_c51d617e-2ed1-40d1-806e-f3cfb0e851e5/postgres/0.log" Sep 11 01:37:37.837827 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:37.837796 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78cd557cc4-ptpp5_39c32d6f-b4ab-4f08-b9d8-2b5fd0998ba5/authorino/0.log" Sep 11 01:37:37.954094 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:37.954065 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-wrxwj_a24001de-37e3-4482-903d-020d8e9cb18e/manager/0.log" Sep 11 01:37:38.298412 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:38.298391 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-sjfvq_1a63eccf-904b-458e-b080-355b92e5bc19/manager/0.log" Sep 11 01:37:38.532291 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:38.532262 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-jzpth_c43cefa7-f77a-49b2-9569-4c27b4b5e68c/manager/0.log" Sep 11 01:37:39.258387 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:39.258356 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cc5ff74fb-xl5vx_ca046967-4442-4ca5-a5b3-85b5867414f4/kube-auth-proxy/0.log" Sep 11 01:37:39.751963 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:39.751929 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5946c9f774-tk9pg_dee19a50-296e-4a89-9e02-21ee8aa3ca87/router/0.log" Sep 11 01:37:40.125281 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:40.125198 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-qqskq_518aded8-c7a6-4bf7-adf7-35117bb089c9/main/0.log" Sep 11 01:37:40.464911 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:40.464885 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-84kdr_1f5a6809-9c04-48b5-9efe-0104d3940ae6/main/0.log" Sep 11 01:37:40.577940 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:40.577916 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpjpfc_bb9982d7-eb58-4c79-af91-56bc80c6c21b/main/0.log" Sep 11 01:37:46.229296 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:46.229265 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:37:46.329980 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:46.329952 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024 Sep 11 01:37:49.342031 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:49.341993 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9rbhz_6038fbb3-56b0-49d0-875a-8f21cfdc9e77/global-pull-secret-syncer/0.log" Sep 11 01:37:49.471485 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:49.471455 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" Sep 11 01:37:49.599414 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:49.599354 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-107.ec2.internal_81a39435d308b5114d55354c422903c5/haproxy/0.log" Sep 11 01:37:53.780089 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:53.780060 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wtd8v_c2b320dd-43e9-435d-8e4f-2bca1f1233ff/ovn-acl-logging/0.log" Sep 11 01:37:53.791399 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:53.791374 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wtd8v_c2b320dd-43e9-435d-8e4f-2bca1f1233ff/ovn-acl-logging/0.log" Sep 11 01:37:54.137230 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:54.137202 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78cd557cc4-ptpp5_39c32d6f-b4ab-4f08-b9d8-2b5fd0998ba5/authorino/0.log" Sep 11 01:37:54.164392 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:54.164372 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-wrxwj_a24001de-37e3-4482-903d-020d8e9cb18e/manager/0.log" Sep 11 01:37:54.260493 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:54.260467 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-sjfvq_1a63eccf-904b-458e-b080-355b92e5bc19/manager/0.log" Sep 11 01:37:54.354557 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:54.354497 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-jzpth_c43cefa7-f77a-49b2-9569-4c27b4b5e68c/manager/0.log" Sep 11 01:37:56.059341 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:56.059312 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-tqxxh_b619e4ba-cf0c-4078-b918-3dbd3f67c119/cluster-monitoring-operator/0.log" Sep 11 01:37:56.301475 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:56.301447 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gwbnp_6236f436-04e9-49e1-8ac1-238b296432a6/node-exporter/0.log" Sep 11 01:37:56.326585 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:56.326521 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gwbnp_6236f436-04e9-49e1-8ac1-238b296432a6/kube-rbac-proxy/0.log" Sep 11 01:37:56.357221 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:56.357197 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gwbnp_6236f436-04e9-49e1-8ac1-238b296432a6/init-textfile/0.log" Sep 11 01:37:58.443024 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:58.442992 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-r7p6n_746635c6-8538-4d95-ad45-7f573caee4b3/networking-console-plugin/0.log" Sep 11 01:37:59.593739 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.593714 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7685ddfb9f-24w22_637e5d57-19bd-4fa5-b4ef-6c3884a5ff43/console/0.log" Sep 11 01:37:59.621807 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.621782 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2"] Sep 11 01:37:59.622222 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.622206 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba4726e-2615-4686-9fb9-f490c7886591" containerName="maas-api" Sep 11 01:37:59.622283 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.622227 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba4726e-2615-4686-9fb9-f490c7886591" containerName="maas-api" Sep 11 01:37:59.622321 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.622304 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ba4726e-2615-4686-9fb9-f490c7886591" containerName="maas-api" Sep 11 01:37:59.626664 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.626644 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2" Sep 11 01:37:59.628897 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.628874 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lvb79\"/\"kube-root-ca.crt\"" Sep 11 01:37:59.629922 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.629881 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lvb79\"/\"default-dockercfg-s54hm\"" Sep 11 01:37:59.629922 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.629888 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lvb79\"/\"openshift-service-ca.crt\"" Sep 11 01:37:59.636304 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.636169 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2"] Sep 11 01:37:59.805421 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.805398 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/84a545cb-d02e-40f2-99d6-404f0b1d7385-podres\") pod \"perf-node-gather-daemonset-w9dp2\" (UID: \"84a545cb-d02e-40f2-99d6-404f0b1d7385\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2" Sep 11 01:37:59.805552 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.805432 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84a545cb-d02e-40f2-99d6-404f0b1d7385-sys\") pod \"perf-node-gather-daemonset-w9dp2\" (UID: \"84a545cb-d02e-40f2-99d6-404f0b1d7385\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2" Sep 11 01:37:59.805552 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.805456 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz9rz\" (UniqueName: \"kubernetes.io/projected/84a545cb-d02e-40f2-99d6-404f0b1d7385-kube-api-access-bz9rz\") pod \"perf-node-gather-daemonset-w9dp2\" (UID: \"84a545cb-d02e-40f2-99d6-404f0b1d7385\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2" Sep 11 01:37:59.805552 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.805475 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84a545cb-d02e-40f2-99d6-404f0b1d7385-lib-modules\") pod \"perf-node-gather-daemonset-w9dp2\" (UID: \"84a545cb-d02e-40f2-99d6-404f0b1d7385\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2" Sep 11 01:37:59.805700 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.805560 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/84a545cb-d02e-40f2-99d6-404f0b1d7385-proc\") pod \"perf-node-gather-daemonset-w9dp2\" (UID: \"84a545cb-d02e-40f2-99d6-404f0b1d7385\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2" Sep 11 01:37:59.906758 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.906696 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/84a545cb-d02e-40f2-99d6-404f0b1d7385-podres\") pod \"perf-node-gather-daemonset-w9dp2\" (UID: \"84a545cb-d02e-40f2-99d6-404f0b1d7385\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2" Sep 11 01:37:59.906758 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.906725 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84a545cb-d02e-40f2-99d6-404f0b1d7385-sys\") pod \"perf-node-gather-daemonset-w9dp2\" (UID: \"84a545cb-d02e-40f2-99d6-404f0b1d7385\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2" Sep 11 01:37:59.906758 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.906745 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz9rz\" (UniqueName: \"kubernetes.io/projected/84a545cb-d02e-40f2-99d6-404f0b1d7385-kube-api-access-bz9rz\") pod \"perf-node-gather-daemonset-w9dp2\" (UID: \"84a545cb-d02e-40f2-99d6-404f0b1d7385\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2" Sep 11 01:37:59.907026 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.906813 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84a545cb-d02e-40f2-99d6-404f0b1d7385-sys\") pod \"perf-node-gather-daemonset-w9dp2\" (UID: \"84a545cb-d02e-40f2-99d6-404f0b1d7385\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2" Sep 11 01:37:59.907026 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.906871 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84a545cb-d02e-40f2-99d6-404f0b1d7385-lib-modules\") pod \"perf-node-gather-daemonset-w9dp2\" (UID: \"84a545cb-d02e-40f2-99d6-404f0b1d7385\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2" Sep 11 01:37:59.907026 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.906890 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/84a545cb-d02e-40f2-99d6-404f0b1d7385-podres\") pod \"perf-node-gather-daemonset-w9dp2\" (UID: \"84a545cb-d02e-40f2-99d6-404f0b1d7385\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2" Sep 11 01:37:59.907026 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.906916 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/84a545cb-d02e-40f2-99d6-404f0b1d7385-proc\") pod \"perf-node-gather-daemonset-w9dp2\" (UID: \"84a545cb-d02e-40f2-99d6-404f0b1d7385\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2" Sep 11 01:37:59.907026 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.907004 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/84a545cb-d02e-40f2-99d6-404f0b1d7385-proc\") pod \"perf-node-gather-daemonset-w9dp2\" (UID: \"84a545cb-d02e-40f2-99d6-404f0b1d7385\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2" Sep 11 01:37:59.907026 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.907009 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84a545cb-d02e-40f2-99d6-404f0b1d7385-lib-modules\") pod \"perf-node-gather-daemonset-w9dp2\" (UID: \"84a545cb-d02e-40f2-99d6-404f0b1d7385\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2" Sep 11 01:37:59.914676 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.914650 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz9rz\" (UniqueName: \"kubernetes.io/projected/84a545cb-d02e-40f2-99d6-404f0b1d7385-kube-api-access-bz9rz\") pod \"perf-node-gather-daemonset-w9dp2\" (UID: \"84a545cb-d02e-40f2-99d6-404f0b1d7385\") " pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2" Sep 11 01:37:59.940163 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:37:59.940142 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2" Sep 11 01:38:00.210415 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:00.210372 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2"] Sep 11 01:38:00.213407 ip-10-0-135-107 kubenswrapper[2554]: W0911 01:38:00.213376 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod84a545cb_d02e_40f2_99d6_404f0b1d7385.slice/crio-0ff8ba2e52f3d80299622e153b8f72ec4f67a82f412912f6937dbc2b5db3408c WatchSource:0}: Error finding container 0ff8ba2e52f3d80299622e153b8f72ec4f67a82f412912f6937dbc2b5db3408c: Status 404 returned error can't find the container with id 0ff8ba2e52f3d80299622e153b8f72ec4f67a82f412912f6937dbc2b5db3408c Sep 11 01:38:00.352925 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:00.352893 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2" event={"ID":"84a545cb-d02e-40f2-99d6-404f0b1d7385","Type":"ContainerStarted","Data":"ffbbd6d5063a896a0e3a3aaf969fb9d17b21c0e7bf8d826e241209f771b51057"} Sep 11 01:38:00.353048 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:00.352968 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2" event={"ID":"84a545cb-d02e-40f2-99d6-404f0b1d7385","Type":"ContainerStarted","Data":"0ff8ba2e52f3d80299622e153b8f72ec4f67a82f412912f6937dbc2b5db3408c"} Sep 11 01:38:00.353048 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:00.353021 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2" Sep 11 01:38:00.370182 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:00.370142 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2" podStartSLOduration=1.370128948 podStartE2EDuration="1.370128948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 01:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 01:38:00.368342195 +0000 UTC m=+2407.191795707" watchObservedRunningTime="2026-09-11 01:38:00.370128948 +0000 UTC m=+2407.193582459" Sep 11 01:38:00.377010 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:00.376994 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-5n9qs_9ba1ed98-f643-461d-92df-c6319ad47f11/volume-data-source-validator/0.log" Sep 11 01:38:01.508076 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:01.508046 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vbxc5_1bd830e3-7d02-4637-b2b7-cf06baad6a26/dns/0.log" Sep 11 01:38:01.527157 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:01.527134 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vbxc5_1bd830e3-7d02-4637-b2b7-cf06baad6a26/kube-rbac-proxy/0.log" Sep 11 01:38:01.638789 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:01.638765 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qxf8s_35c8be8a-e0b7-4843-b8db-4b6a486aee9e/dns-node-resolver/0.log" Sep 11 01:38:02.273245 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:02.273214 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-spz6s_70820178-37d7-4c9e-8240-0905cdfe37ee/node-ca/0.log" Sep 11 01:38:03.491577 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:03.491545 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cc5ff74fb-xl5vx_ca046967-4442-4ca5-a5b3-85b5867414f4/kube-auth-proxy/0.log" Sep 11 01:38:03.600765 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:03.600739 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5946c9f774-tk9pg_dee19a50-296e-4a89-9e02-21ee8aa3ca87/router/0.log" Sep 11 01:38:04.521474 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:04.521441 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ffbjw_a04fe26a-2690-4d67-bd7a-c19c9d610589/serve-healthcheck-canary/0.log" Sep 11 01:38:04.975309 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:04.975260 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2fhzl_e048904e-6a28-4df4-8d94-a50b3e15e161/kube-rbac-proxy/0.log" Sep 11 01:38:04.994296 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:04.994269 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2fhzl_e048904e-6a28-4df4-8d94-a50b3e15e161/exporter/0.log" Sep 11 01:38:05.014219 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:05.014199 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2fhzl_e048904e-6a28-4df4-8d94-a50b3e15e161/extractor/0.log" Sep 11 01:38:06.368383 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:06.368357 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lvb79/perf-node-gather-daemonset-w9dp2" Sep 11 01:38:07.245796 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:07.245759 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-kfzfx_7ddea17b-8323-4ad1-8171-a6e6e00f3133/manager/0.log" Sep 11 01:38:07.489863 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:07.489815 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-4ktqt_0cf5f7e0-6bb4-4349-9806-42540500a8d2/manager/0.log" Sep 11 01:38:07.556838 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:07.556774 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-6mqjx_412594fe-d273-4015-87ae-48e593f0f257/server/0.log" Sep 11 01:38:07.815738 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:07.815633 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c49b45946-qpjtx_5115bd54-60ec-4cd8-9e8e-06436e61adf0/manager/0.log" Sep 11 01:38:07.838457 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:07.838435 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c49b45946-qpjtx_5115bd54-60ec-4cd8-9e8e-06436e61adf0/copy-manifests/0.log" Sep 11 01:38:09.261763 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:09.261707 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-mn7v6_c17083fd-fc2d-4fe4-a34d-963714a3e42c/openshift-lws-operator/0.log" Sep 11 01:38:10.214886 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:10.214835 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cc5ff74fb-xl5vx_ca046967-4442-4ca5-a5b3-85b5867414f4/kube-auth-proxy/0.log" Sep 11 01:38:10.291268 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:10.291239 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5946c9f774-tk9pg_dee19a50-296e-4a89-9e02-21ee8aa3ca87/router/0.log" Sep 11 01:38:12.150353 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:12.150323 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cc5ff74fb-xl5vx_ca046967-4442-4ca5-a5b3-85b5867414f4/kube-auth-proxy/0.log" Sep 11 01:38:12.231025 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:12.230996 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5946c9f774-tk9pg_dee19a50-296e-4a89-9e02-21ee8aa3ca87/router/0.log" Sep 11 01:38:14.704087 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:14.704050 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-mng4p_2145b039-e727-4271-ad8e-e15b459b2a24/kube-storage-version-migrator-operator/1.log" Sep 11 01:38:14.704730 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:14.704708 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-mng4p_2145b039-e727-4271-ad8e-e15b459b2a24/kube-storage-version-migrator-operator/0.log" Sep 11 01:38:15.931497 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:15.931469 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kt8nl_d5568658-3c63-4779-86b5-df6dc73de2c7/kube-multus-additional-cni-plugins/0.log" Sep 11 01:38:15.952301 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:15.952276 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kt8nl_d5568658-3c63-4779-86b5-df6dc73de2c7/egress-router-binary-copy/0.log" Sep 11 01:38:15.971523 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:15.971504 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kt8nl_d5568658-3c63-4779-86b5-df6dc73de2c7/cni-plugins/0.log" Sep 11 01:38:15.992155 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:15.992138 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kt8nl_d5568658-3c63-4779-86b5-df6dc73de2c7/bond-cni-plugin/0.log" Sep 11 01:38:16.013746 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:16.013700 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kt8nl_d5568658-3c63-4779-86b5-df6dc73de2c7/routeoverride-cni/0.log" Sep 11 01:38:16.033167 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:16.033150 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kt8nl_d5568658-3c63-4779-86b5-df6dc73de2c7/whereabouts-cni-bincopy/0.log" Sep 11 01:38:16.054562 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:16.054542 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kt8nl_d5568658-3c63-4779-86b5-df6dc73de2c7/whereabouts-cni/0.log" Sep 11 01:38:16.231170 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:16.231111 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bzv5j_ac49dda0-6ce3-42d9-83e7-1e22ffa2a096/kube-multus/0.log" Sep 11 01:38:16.296122 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:16.296096 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-87kmh_8f0c9d36-ae86-4625-8dd1-0c9764c655e5/network-metrics-daemon/0.log" Sep 11 01:38:16.313463 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:16.313443 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-87kmh_8f0c9d36-ae86-4625-8dd1-0c9764c655e5/kube-rbac-proxy/0.log" Sep 11 01:38:16.559440 ip-10-0-135-107 kubenswrapper[2554]: I0911 01:38:16.559379 2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-45g5g_3794d977-c81b-4042-ad21-bd34d7462018/konnectivity-agent/0.log" limit=1024