Sep 11 17:43:11.866696 ip-10-0-138-96 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 17:43:11.866709 ip-10-0-138-96 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 17:43:11.866719 ip-10-0-138-96 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 17:43:11.866933 ip-10-0-138-96 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 17:43:21.980496 ip-10-0-138-96 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 17:43:21.980526 ip-10-0-138-96 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot a5923d5d8527474fb3e85295f21afb61 --
Sep 11 17:44:54.427662 ip-10-0-138-96 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 17:44:54.844474 ip-10-0-138-96 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 17:44:54.844474 ip-10-0-138-96 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 17:44:54.844474 ip-10-0-138-96 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 17:44:54.844474 ip-10-0-138-96 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 17:44:54.844474 ip-10-0-138-96 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 17:44:54.845136 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.844462    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 17:44:54.847624 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847606    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:44:54.847624 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847623    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:44:54.847624 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847627    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:44:54.847714 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847631    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:44:54.847714 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847635    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:44:54.847714 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847639    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:44:54.847714 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847642    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:44:54.847714 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847645    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:44:54.847714 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847647    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:44:54.847714 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847650    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:44:54.847714 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847653    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:44:54.847714 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847655    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:44:54.847714 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847658    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:44:54.847714 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847666    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:44:54.847714 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847668    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:44:54.847714 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847671    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:44:54.847714 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847674    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:44:54.847714 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847676    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:44:54.847714 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847679    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:44:54.847714 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847681    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:44:54.847714 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847686    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:44:54.847714 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847690    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:44:54.848175 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847693    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:44:54.848175 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847696    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:44:54.848175 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847699    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:44:54.848175 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847702    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:44:54.848175 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847705    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:44:54.848175 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847707    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:44:54.848175 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847711    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:44:54.848175 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847713    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:44:54.848175 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847716    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:44:54.848175 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847719    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:44:54.848175 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847722    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:44:54.848175 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847725    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:44:54.848175 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847728    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:44:54.848175 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847730    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:44:54.848175 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847734    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:44:54.848175 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847738    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:44:54.848175 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847740    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:44:54.848175 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847744    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:44:54.848175 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847746    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:44:54.848660 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847749    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:44:54.848660 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847751    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:44:54.848660 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847754    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:44:54.848660 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847756    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:44:54.848660 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847759    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:44:54.848660 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847762    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:44:54.848660 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847764    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:44:54.848660 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847767    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:44:54.848660 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847769    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:44:54.848660 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847771    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:44:54.848660 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847774    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:44:54.848660 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847776    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:44:54.848660 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847780    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:44:54.848660 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847782    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:44:54.848660 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847785    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:44:54.848660 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847788    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:44:54.848660 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847791    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:44:54.848660 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847793    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:44:54.848660 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847796    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:44:54.848660 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847799    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:44:54.849181 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847802    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:44:54.849181 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847805    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:44:54.849181 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847808    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:44:54.849181 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847810    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:44:54.849181 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847813    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:44:54.849181 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847815    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:44:54.849181 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847818    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:44:54.849181 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847820    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:44:54.849181 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847823    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:44:54.849181 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847825    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:44:54.849181 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847829    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:44:54.849181 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847833    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:44:54.849181 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847835    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:44:54.849181 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847838    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:44:54.849181 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847841    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:44:54.849181 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847843    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:44:54.849181 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847846    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:44:54.849181 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847849    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:44:54.849181 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847852    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:44:54.849181 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847854    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:44:54.849689 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847857    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:44:54.849689 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847859    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:44:54.849689 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847862    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:44:54.849689 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847864    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:44:54.849689 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847867    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:44:54.849689 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847870    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:44:54.849689 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847872    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:44:54.849689 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847875    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:44:54.849689 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847878    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:44:54.849689 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847881    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:44:54.849689 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847884    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:44:54.849689 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847887    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:44:54.849689 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847890    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:44:54.849689 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847892    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:44:54.849689 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847896    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:44:54.849689 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847899    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:44:54.849689 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847902    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:44:54.849689 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.847905    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:44:54.849689 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848620    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:44:54.850146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848627    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:44:54.850146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848630    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:44:54.850146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848634    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:44:54.850146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848636    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:44:54.850146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848640    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:44:54.850146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848643    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:44:54.850146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848645    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:44:54.850146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848648    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:44:54.850146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848651    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:44:54.850146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848653    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:44:54.850146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848656    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:44:54.850146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848660    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:44:54.850146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848662    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:44:54.850146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848665    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:44:54.850146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848668    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:44:54.850146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848670    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:44:54.850146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848673    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:44:54.850146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848676    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:44:54.850146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848679    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:44:54.850146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848682    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:44:54.850667 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848685    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:44:54.850667 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848687    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:44:54.850667 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848690    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:44:54.850667 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848693    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:44:54.850667 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848696    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:44:54.850667 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848699    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:44:54.850667 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848701    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:44:54.850667 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848704    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:44:54.850667 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848707    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:44:54.850667 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848709    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:44:54.850667 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848712    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:44:54.850667 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848715    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:44:54.850667 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848717    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:44:54.850667 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848720    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:44:54.850667 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848722    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:44:54.850667 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848725    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:44:54.850667 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848728    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:44:54.850667 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848731    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:44:54.850667 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848733    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:44:54.850667 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848736    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:44:54.851152 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848738    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:44:54.851152 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848741    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:44:54.851152 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848743    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:44:54.851152 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848746    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:44:54.851152 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848748    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:44:54.851152 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848751    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:44:54.851152 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848753    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:44:54.851152 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848756    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:44:54.851152 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848758    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:44:54.851152 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848761    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:44:54.851152 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848763    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:44:54.851152 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848767    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:44:54.851152 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848770    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:44:54.851152 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848772    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:44:54.851152 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848775    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:44:54.851152 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848777    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:44:54.851152 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848781    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:44:54.851152 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848784    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:44:54.851152 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848786    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:44:54.851152 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848789    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:44:54.851664 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848794    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:44:54.851664 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848797    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:44:54.851664 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848799    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:44:54.851664 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848802    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:44:54.851664 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848806    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:44:54.851664 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848810    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:44:54.851664 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848814    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:44:54.851664 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848816    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:44:54.851664 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848820    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:44:54.851664 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848823    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:44:54.851664 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848830    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:44:54.851664 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848833    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:44:54.851664 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848836    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:44:54.851664 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848840    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:44:54.851664 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848843    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:44:54.851664 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848845    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:44:54.851664 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848849    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:44:54.851664 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848853    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:44:54.851664 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848856    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:44:54.852128 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848859    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:44:54.852128 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848862    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:44:54.852128 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848864    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:44:54.852128 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848867    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:44:54.852128 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848870    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:44:54.852128 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848874    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:44:54.852128 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848877    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:44:54.852128 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848879    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:44:54.852128 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848882    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:44:54.852128 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848886    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:44:54.852128 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848890    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:44:54.852128 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848893    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:44:54.852128 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848896    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:44:54.852128 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848900    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:44:54.852128 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848903    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:44:54.852128 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848905    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:44:54.852128 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848908    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:44:54.852128 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848911    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:44:54.852128 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.848913    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:44:54.852128 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849316    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:44:54.852637 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849321    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:44:54.852637 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849324    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:44:54.852637 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849327    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:44:54.852637 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849331    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:44:54.852637 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849334    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:44:54.852637 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849336    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:44:54.852637 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849340    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:44:54.852637 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849343    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:44:54.852637 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849345    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:44:54.852637 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849363    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:44:54.852637 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849366    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:44:54.852637 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849370    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:44:54.852637 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849374    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:44:54.852637 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849377    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:44:54.852637 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849380    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:44:54.852637 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849382    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:44:54.852637 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849386    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:44:54.852637 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849388    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:44:54.852637 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849391    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:44:54.852637 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849394    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:44:54.853122 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849398    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:44:54.853122 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849401    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:44:54.853122 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849404    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:44:54.853122 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849407    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:44:54.853122 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849410    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:44:54.853122 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849414    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:44:54.853122 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849418    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:44:54.853122 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849421    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:44:54.853122 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849424    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:44:54.853122 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849427    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:44:54.853122 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849430    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:44:54.853122 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849433    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:44:54.853122 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849436    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:44:54.853122 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849438    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:44:54.853122 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849442    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:44:54.853122 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849444    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:44:54.853122 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849448    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:44:54.853122 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849450    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:44:54.853122 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849453    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:44:54.853610 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849456    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:44:54.853610 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849458    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:44:54.853610 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849461    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:44:54.853610 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849463    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:44:54.853610 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849466    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:44:54.853610 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849469    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:44:54.853610 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849471    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:44:54.853610 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849474    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:44:54.853610 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849476    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:44:54.853610 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849479    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:44:54.853610 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849482    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:44:54.853610 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849484    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:44:54.853610 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849487    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:44:54.853610 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849489    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:44:54.853610 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849492    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:44:54.853610 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849495    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:44:54.853610 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849497    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:44:54.853610 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849500    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:44:54.853610 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849503    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:44:54.853610 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849505    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:44:54.854138 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849508    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:44:54.854138 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849510    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:44:54.854138 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849513    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:44:54.854138 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849515    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:44:54.854138 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849518    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:44:54.854138 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849520    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:44:54.854138 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849523    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:44:54.854138 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849525    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:44:54.854138 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849529    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:44:54.854138 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849533    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:44:54.854138 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849536    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:44:54.854138 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849539    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:44:54.854138 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849542    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:44:54.854138 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849545    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:44:54.854138 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849549    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:44:54.854138 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849552    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:44:54.854138 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849555    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:44:54.854138 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849557    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:44:54.854138 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849560    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:44:54.854624 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849562    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:44:54.854624 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849565    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:44:54.854624 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849568    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:44:54.854624 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849570    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:44:54.854624 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849573    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:44:54.854624 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849576    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:44:54.854624 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849578    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:44:54.854624 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849583    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:44:54.854624 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849586    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:44:54.854624 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849588    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:44:54.854624 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849591    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:44:54.854624 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849594    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:44:54.854624 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849597    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:44:54.854624 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849599    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:44:54.854624 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849602    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:44:54.854624 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849605    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:44:54.854624 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849607    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:44:54.854624 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849610    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:44:54.854624 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849612    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:44:54.854624 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.849615    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:44:54.855106 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849693    2559 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 17:44:54.855106 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849728    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 17:44:54.855106 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849737    2559 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 17:44:54.855106 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849741    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 17:44:54.855106 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849746    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 17:44:54.855106 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849750    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 17:44:54.855106 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849754    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 17:44:54.855106 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849759    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 17:44:54.855106 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849762    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 17:44:54.855106 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849765    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 17:44:54.855106 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849769    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 17:44:54.855106 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849772    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 17:44:54.855106 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849776    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 17:44:54.855106 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849779    2559 flags.go:64] FLAG: --cgroup-root=""
Sep 11 17:44:54.855106 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849782    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 17:44:54.855106 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849785    2559 flags.go:64] FLAG: --client-ca-file=""
Sep 11 17:44:54.855106 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849788    2559 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 17:44:54.855106 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849791    2559 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 17:44:54.855106 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849796    2559 flags.go:64] FLAG: --cluster-domain=""
Sep 11 17:44:54.855106 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849800    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 17:44:54.855106 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849803    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 17:44:54.855106 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849807    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 17:44:54.855106 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849810    2559 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 17:44:54.855106 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849814    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849817    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849820    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849824    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849827    2559 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849830    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849833    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849837    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849840    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849844    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849847    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849851    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849854    2559 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849858    2559 flags.go:64] FLAG: --enable-server="true"
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849861    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849866    2559 flags.go:64] FLAG: --event-burst="100"
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849869    2559 flags.go:64] FLAG: --event-qps="50"
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849872    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849875    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849878    2559 flags.go:64] FLAG: --eviction-hard=""
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849882    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849885    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849888    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849891    2559 flags.go:64] FLAG: --eviction-soft=""
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849894    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 17:44:54.855688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849897    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 17:44:54.856300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849901    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 17:44:54.856300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849904    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 17:44:54.856300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849908    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 17:44:54.856300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849911    2559 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 17:44:54.856300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849914    2559 flags.go:64] FLAG: --feature-gates=""
Sep 11 17:44:54.856300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849918    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 17:44:54.856300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849922    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 17:44:54.856300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849925    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 17:44:54.856300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849928    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 17:44:54.856300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849931    2559 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 17:44:54.856300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849934    2559 flags.go:64] FLAG: --help="false"
Sep 11 17:44:54.856300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849937    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-138-96.ec2.internal"
Sep 11 17:44:54.856300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849940    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 17:44:54.856300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849943    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 17:44:54.856300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849946    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 17:44:54.856300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849950    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 17:44:54.856300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849953    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 17:44:54.856300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849958    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 17:44:54.856300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849961    2559 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 17:44:54.856300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849964    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 17:44:54.856300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849967    2559 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 17:44:54.856300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849970    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 17:44:54.856300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849973    2559 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 17:44:54.856300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849976    2559 flags.go:64] FLAG: --kube-reserved=""
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849980    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849983    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849986    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849989    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849992    2559 flags.go:64] FLAG: --lock-file=""
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849995    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.849998    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850001    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850006    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850009    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850013    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850016    2559 flags.go:64] FLAG: --logging-format="text"
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850019    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850023    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850026    2559 flags.go:64] FLAG: --manifest-url=""
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850028    2559 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850033    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850036    2559 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850040    2559 flags.go:64] FLAG: --max-pods="110"
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850043    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850046    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850049    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850052    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850055    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 17:44:54.856921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850058    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850063    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850070    2559 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850074    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850077    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850080    2559 flags.go:64] FLAG: --pod-cidr=""
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850087    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850093    2559 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850096    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850099    2559 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850102    2559 flags.go:64] FLAG: --port="10250"
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850105    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850108    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0701bd4ff5e06cc49"
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850111    2559 flags.go:64] FLAG: --qos-reserved=""
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850114    2559 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850117    2559 flags.go:64] FLAG: --register-node="true"
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850120    2559 flags.go:64] FLAG: --register-with-taints=""
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850124    2559 flags.go:64] FLAG: --registry-burst="10"
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850128    2559 flags.go:64] FLAG: --registry-qps="5"
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850131    2559 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850134    2559 flags.go:64] FLAG: --reserved-memory=""
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850138    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850140    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850144    2559 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850147    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 17:44:54.858017 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850150    2559 flags.go:64] FLAG: --runonce="false"
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850153    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850156    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850159    2559 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850163    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850165    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850168    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850171    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850175    2559 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850180    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850183    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850186    2559 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850189    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850192    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850195    2559 flags.go:64] FLAG: --system-cgroups=""
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850198    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850203    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850206    2559 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850209    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850214    2559 flags.go:64] FLAG: --tls-min-version=""
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850217    2559 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850220    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850223    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850226    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850229    2559 flags.go:64] FLAG: --v="2"
Sep 11 17:44:54.859112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850234    2559 flags.go:64] FLAG: --version="false"
Sep 11 17:44:54.859949 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850238    2559 flags.go:64] FLAG: --vmodule=""
Sep 11 17:44:54.859949 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850242    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 17:44:54.859949 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850245    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 17:44:54.859949 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850378    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:44:54.859949 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850394    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:44:54.859949 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850398    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:44:54.859949 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850402    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:44:54.859949 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850406    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:44:54.859949 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850410    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:44:54.859949 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850413    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:44:54.859949 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850416    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:44:54.859949 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850418    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:44:54.859949 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850421    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:44:54.859949 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850424    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:44:54.859949 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850427    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:44:54.859949 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850431    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:44:54.859949 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850433    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:44:54.859949 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850436    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:44:54.859949 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850438    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:44:54.859949 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850441    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:44:54.860493 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850444    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:44:54.860493 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850447    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:44:54.860493 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850450    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:44:54.860493 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850453    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:44:54.860493 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850456    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:44:54.860493 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850458    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:44:54.860493 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850461    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:44:54.860493 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850464    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:44:54.860493 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850467    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:44:54.860493 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850470    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:44:54.860493 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850474    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:44:54.860493 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850478    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:44:54.860493 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850482    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:44:54.860493 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850485    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:44:54.860493 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850487    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:44:54.860493 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850490    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:44:54.860493 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850493    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:44:54.860493 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850495    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:44:54.860493 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850498    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:44:54.860957 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850501    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:44:54.860957 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850504    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:44:54.860957 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850506    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:44:54.860957 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850509    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:44:54.860957 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850511    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:44:54.860957 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850514    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:44:54.860957 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850516    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:44:54.860957 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850519    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:44:54.860957 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850523    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:44:54.860957 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850526    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:44:54.860957 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850528    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:44:54.860957 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850531    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:44:54.860957 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850534    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:44:54.860957 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850536    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:44:54.860957 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850539    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:44:54.860957 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850542    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:44:54.860957 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850545    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:44:54.860957 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850549    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:44:54.860957 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850552    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:44:54.861597 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850555    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:44:54.861597 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850558    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:44:54.861597 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850561    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:44:54.861597 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850564    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:44:54.861597 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850566    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:44:54.861597 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850569    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:44:54.861597 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850574    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:44:54.861597 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850576    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:44:54.861597 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850579    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:44:54.861597 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850582    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:44:54.861597 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850584    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:44:54.861597 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850587    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:44:54.861597 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850590    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:44:54.861597 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850592    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:44:54.861597 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850595    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:44:54.861597 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850598    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:44:54.861597 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850600    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:44:54.861597 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850603    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:44:54.861597 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850605    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:44:54.861597 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850608    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:44:54.862482 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850611    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:44:54.862482 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850615    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:44:54.862482 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850617    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:44:54.862482 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850621    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:44:54.862482 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850623    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:44:54.862482 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850626    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:44:54.862482 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850629    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:44:54.862482 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850631    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:44:54.862482 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850634    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:44:54.862482 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850637    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:44:54.862482 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850640    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:44:54.862482 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850643    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:44:54.862482 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850666    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:44:54.862482 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850670    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:44:54.862482 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850673    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:44:54.862482 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850676    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:44:54.862482 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850679    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:44:54.862482 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850682    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:44:54.862482 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850685    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:44:54.862482 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850689    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:44:54.863146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850692    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:44:54.863146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850695    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:44:54.863146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850698    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:44:54.863146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.850700    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:44:54.863146 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.850705    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 17:44:54.863146 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.859307    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 17:44:54.863146 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.859329    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 17:44:54.863146 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.861275    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 17:44:54.863146 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.861294    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 17:44:54.863146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861404    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:44:54.863146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861413    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:44:54.863146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861420    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:44:54.863146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861425    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:44:54.863146 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861429    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:44:54.863607 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861434    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:44:54.863607 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861438    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:44:54.863607 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861444    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:44:54.863607 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861449    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:44:54.863607 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861454    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:44:54.863607 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861459    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:44:54.863607 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861466    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:44:54.863607 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861472    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:44:54.863607 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861477    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:44:54.863607 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861481    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:44:54.863607 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861486    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:44:54.863607 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861490    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:44:54.863607 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861494    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:44:54.863607 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861499    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:44:54.863607 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861503    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:44:54.863607 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861507    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:44:54.863607 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861511    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:44:54.863607 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861515    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:44:54.863607 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861520    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:44:54.864540 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861524    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:44:54.864540 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861530    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:44:54.864540 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861535    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:44:54.864540 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861539    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:44:54.864540 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861543    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:44:54.864540 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861556    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:44:54.864540 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861561    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:44:54.864540 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861565    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:44:54.864540 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861569    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:44:54.864540 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861574    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:44:54.864540 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861578    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:44:54.864540 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861582    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:44:54.864540 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861586    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:44:54.864540 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861591    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:44:54.864540 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861595    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:44:54.864540 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861599    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:44:54.864540 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861603    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:44:54.864540 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861607    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:44:54.864540 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861610    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:44:54.864540 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861614    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:44:54.865190 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861618    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:44:54.865190 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861622    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:44:54.865190 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861626    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:44:54.865190 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861631    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:44:54.865190 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861635    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:44:54.865190 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861640    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:44:54.865190 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861644    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:44:54.865190 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861648    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:44:54.865190 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861653    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:44:54.865190 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861657    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:44:54.865190 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861661    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:44:54.865190 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861665    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:44:54.865190 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861669    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:44:54.865190 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861674    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:44:54.865190 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861678    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:44:54.865190 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861683    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:44:54.865190 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861688    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:44:54.865190 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861692    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:44:54.865190 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861698    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:44:54.865828 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861702    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:44:54.865828 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861707    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:44:54.865828 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861711    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:44:54.865828 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861715    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:44:54.865828 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861720    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:44:54.865828 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861724    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:44:54.865828 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861728    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:44:54.865828 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861733    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:44:54.865828 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861739    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:44:54.865828 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861744    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:44:54.865828 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861749    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:44:54.865828 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861753    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:44:54.865828 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861757    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:44:54.865828 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861761    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:44:54.865828 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861765    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:44:54.865828 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861770    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:44:54.865828 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861775    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:44:54.865828 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861780    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:44:54.865828 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861784    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:44:54.865828 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861788    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:44:54.866311 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861793    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:44:54.866311 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861796    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:44:54.866311 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861801    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:44:54.866311 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861806    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:44:54.866311 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861810    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:44:54.866311 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861814    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:44:54.866311 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861818    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:44:54.866311 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861823    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:44:54.866311 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861827    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:44:54.866311 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861831    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:44:54.866311 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861835    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:44:54.866311 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861840    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:44:54.866311 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861845    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:44:54.866311 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861849    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:44:54.866311 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861853    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:44:54.866311 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.861860    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:44:54.866746 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.861868    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 17:44:54.866746 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862057    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:44:54.866746 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862066    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:44:54.866746 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862070    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:44:54.866746 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862075    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:44:54.866746 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862080    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:44:54.866746 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862085    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:44:54.866746 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862089    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:44:54.866746 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862094    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:44:54.866746 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862100    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:44:54.866746 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862106    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:44:54.866746 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862111    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:44:54.866746 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862116    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:44:54.866746 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862121    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:44:54.866746 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862126    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:44:54.867149 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862130    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:44:54.867149 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862134    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:44:54.867149 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862138    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:44:54.867149 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862143    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:44:54.867149 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862147    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:44:54.867149 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862151    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:44:54.867149 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862155    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:44:54.867149 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862159    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:44:54.867149 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862165    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:44:54.867149 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862169    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:44:54.867149 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862173    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:44:54.867149 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862178    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:44:54.867149 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862183    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:44:54.867149 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862187    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:44:54.867149 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862192    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:44:54.867149 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862196    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:44:54.867149 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862200    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:44:54.867149 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862206    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:44:54.867149 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862212    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:44:54.867668 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862217    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:44:54.867668 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862221    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:44:54.867668 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862225    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:44:54.867668 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862229    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:44:54.867668 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862234    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:44:54.867668 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862238    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:44:54.867668 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862242    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:44:54.867668 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862246    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:44:54.867668 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862251    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:44:54.867668 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862255    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:44:54.867668 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862259    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:44:54.867668 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862263    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:44:54.867668 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862268    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:44:54.867668 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862274    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:44:54.867668 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862280    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:44:54.867668 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862284    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:44:54.867668 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862289    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:44:54.867668 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862294    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:44:54.867668 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862298    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:44:54.867668 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862303    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:44:54.868159 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862307    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:44:54.868159 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862313    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:44:54.868159 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862318    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:44:54.868159 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862322    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:44:54.868159 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862326    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:44:54.868159 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862331    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:44:54.868159 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862335    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:44:54.868159 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862339    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:44:54.868159 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862345    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:44:54.868159 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862366    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:44:54.868159 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862370    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:44:54.868159 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862375    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:44:54.868159 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862379    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:44:54.868159 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862384    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:44:54.868159 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862388    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:44:54.868159 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862392    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:44:54.868159 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862396    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:44:54.868159 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862400    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:44:54.868159 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862404    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:44:54.868753 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862409    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:44:54.868753 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862414    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:44:54.868753 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862418    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:44:54.868753 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862422    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:44:54.868753 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862426    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:44:54.868753 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862430    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:44:54.868753 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862435    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:44:54.868753 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862439    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:44:54.868753 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862443    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:44:54.868753 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862447    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:44:54.868753 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862451    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:44:54.868753 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862456    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:44:54.868753 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862460    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:44:54.868753 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862465    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:44:54.868753 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862469    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:44:54.868753 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862473    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:44:54.868753 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862478    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:44:54.868753 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862482    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:44:54.868753 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862486    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:44:54.868753 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862490    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:44:54.869292 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862494    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:44:54.869292 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862498    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:44:54.869292 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862503    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:44:54.869292 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862507    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:44:54.869292 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862511    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:44:54.869292 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862516    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:44:54.869292 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:54.862520    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:44:54.869292 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.862529    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 17:44:54.869292 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.862705    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 17:44:54.869292 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.865582    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 17:44:54.869292 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.866712    2559 server.go:1019] "Starting client certificate rotation"
Sep 11 17:44:54.869292 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.866818    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 17:44:54.869292 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.867670    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 17:44:54.869954 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.869932    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 17:44:54.871403 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.871338    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 17:44:54.889262 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.889240    2559 log.go:25] "Validated CRI v1 runtime API"
Sep 11 17:44:54.893440 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.893426    2559 log.go:25] "Validated CRI v1 image API"
Sep 11 17:44:54.894275 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.894260    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 17:44:54.898532 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.898514    2559 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7e040e4b-9687-41f4-bb85-baa7dfe24a91:/dev/nvme0n1p3 fbabb590-7cbc-4996-a303-fbd1fe7eca4f:/dev/nvme0n1p4]
Sep 11 17:44:54.898575 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.898532    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 17:44:54.902065 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.902042    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 17:44:54.904962 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.904842    2559 manager.go:216] Machine: {Timestamp:2026-09-11 17:44:54.903492891 +0000 UTC m=+0.348593453 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100345 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec246836fafa65a4fbf5582c3e308b30 SystemUUID:ec246836-fafa-65a4-fbf5-582c3e308b30 BootID:a5923d5d-8527-474f-b3e8-5295f21afb61 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e1:fb:55:1a:a5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e1:fb:55:1a:a5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:70:7e:34:43:d6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 17:44:54.904962 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.904958    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 17:44:54.905090 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.905041    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 17:44:54.908600 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.908574    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 17:44:54.908747 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.908602    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-96.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 17:44:54.909442 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.909432    2559 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 17:44:54.909479 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.909444    2559 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 17:44:54.909479 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.909451    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 17:44:54.910255 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.910245    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 17:44:54.911455 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.911443    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 17:44:54.916616 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.916604    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 17:44:54.916740 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.916730    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 17:44:54.916772 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.916764    2559 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 17:44:54.916799 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.916778    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 17:44:54.916829 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.916801    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 17:44:54.916829 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.916817    2559 kubelet.go:398] "Adding apiserver pod source"
Sep 11 17:44:54.916887 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.916830    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 17:44:54.917277 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.917255    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 17:44:54.917407 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.917280    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 17:44:54.922515 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.922495    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 17:44:54.925731 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.925712    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 17:44:54.925828 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.925745    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 17:44:54.927800 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.927788    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 17:44:54.927868 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.927804    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 17:44:54.927868 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.927810    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 17:44:54.927868 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.927815    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 17:44:54.927868 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.927823    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 17:44:54.927868 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.927829    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 17:44:54.927868 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.927842    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 17:44:54.927868 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.927847    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 17:44:54.927868 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.927853    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 17:44:54.927868 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.927858    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 17:44:54.927868 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.927866    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 17:44:54.927868 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.927872    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 17:44:54.928773 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.928762    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 17:44:54.928773 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.928772    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 17:44:54.929176 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:54.929138    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 17:44:54.929176 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:54.929142    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-96.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 17:44:54.932452 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.932441    2559 server.go:1268] "Started kubelet"
Sep 11 17:44:54.932532 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.932501    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 17:44:54.932623 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.932553    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 17:44:54.932672 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.932625    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 17:44:54.933233 ip-10-0-138-96 systemd[1]: Started Kubernetes Kubelet.
Sep 11 17:44:54.934344 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.934326    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 17:44:54.935184 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.935170    2559 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 17:44:54.937074 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:54.937055    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 17:44:54.940306 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.940290    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-96.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 17:44:54.941249 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:54.940174    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-96.ec2.internal.18d455423bcce52c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-96.ec2.internal,UID:ip-10-0-138-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-96.ec2.internal,},FirstTimestamp:2026-09-11 17:44:54.932415788 +0000 UTC m=+0.377516350,LastTimestamp:2026-09-11 17:44:54.932415788 +0000 UTC m=+0.377516350,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-96.ec2.internal,}"
Sep 11 17:44:54.941837 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.941822    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 17:44:54.941901 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.941839    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 17:44:54.941968 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:54.941940    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-96.ec2.internal\" not found"
Sep 11 17:44:54.942602 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.942587    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 17:44:54.942674 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.942588    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 17:44:54.942674 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.942626    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 17:44:54.942762 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.942730    2559 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 17:44:54.942762 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.942738    2559 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 17:44:54.943907 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.943890    2559 factory.go:55] Registering systemd factory
Sep 11 17:44:54.943907 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.943906    2559 factory.go:223] Registration of the systemd container factory successfully
Sep 11 17:44:54.944107 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.944097    2559 factory.go:153] Registering CRI-O factory
Sep 11 17:44:54.944107 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.944108    2559 factory.go:223] Registration of the crio container factory successfully
Sep 11 17:44:54.944178 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.944159    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 17:44:54.944220 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.944178    2559 factory.go:103] Registering Raw factory
Sep 11 17:44:54.944220 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.944193    2559 manager.go:1196] Started watching for new ooms in manager
Sep 11 17:44:54.944618 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.944606    2559 manager.go:349] Starting recovery of all containers
Sep 11 17:44:54.945703 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:54.945647    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-96.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 17:44:54.946076 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:54.946056    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 17:44:54.957491 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.957471    2559 manager.go:354] Recovery completed
Sep 11 17:44:54.961294 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.961270    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qjcqq"
Sep 11 17:44:54.962253 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.962242    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:44:54.964018 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.963995    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:44:54.964108 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.964036    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:44:54.964108 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.964049    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:44:54.964590 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.964573    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 17:44:54.964590 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.964588    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 17:44:54.964681 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.964592    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:44:54.964681 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.964606    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 17:44:54.965250 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.965238    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:44:54.965290 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.965261    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:44:54.965290 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.965270    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:44:54.965913 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:54.965844    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-96.ec2.internal.18d455423daf1bb8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-96.ec2.internal,UID:ip-10-0-138-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-96.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-96.ec2.internal,},FirstTimestamp:2026-09-11 17:44:54.964018104 +0000 UTC m=+0.409118675,LastTimestamp:2026-09-11 17:44:54.964018104 +0000 UTC m=+0.409118675,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-96.ec2.internal,}"
Sep 11 17:44:54.966667 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.966656    2559 policy_none.go:49] "None policy: Start"
Sep 11 17:44:54.966712 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.966671    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 17:44:54.966712 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.966680    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 17:44:54.968680 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.968668    2559 policy_none.go:47] "Start"
Sep 11 17:44:54.969763 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.969746    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qjcqq"
Sep 11 17:44:54.975702 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:54.975615    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-96.ec2.internal.18d455423daf7c73  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-96.ec2.internal,UID:ip-10-0-138-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-96.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-96.ec2.internal,},FirstTimestamp:2026-09-11 17:44:54.964042867 +0000 UTC m=+0.409143429,LastTimestamp:2026-09-11 17:44:54.964042867 +0000 UTC m=+0.409143429,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-96.ec2.internal,}"
Sep 11 17:44:54.997990 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.997975    2559 manager.go:337] "Starting Device Plugin manager"
Sep 11 17:44:55.023339 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:54.998016    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 17:44:55.023339 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.998028    2559 server.go:87] "Starting device plugin registration server"
Sep 11 17:44:55.023339 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.998298    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 17:44:55.023339 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.998309    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 17:44:55.023339 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.998445    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 17:44:55.023339 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.998514    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 17:44:55.023339 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:54.998523    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 17:44:55.023339 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:54.999010    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 17:44:55.023339 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:54.999054    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-96.ec2.internal\" not found"
Sep 11 17:44:55.044997 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.044962    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 17:44:55.046177 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.046160    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 17:44:55.046253 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.046218    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 17:44:55.046253 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.046245    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 17:44:55.046336 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:55.046289    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 17:44:55.049005 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.048986    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:44:55.099185 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.099095    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:44:55.100186 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.100168    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:44:55.100301 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.100197    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:44:55.100301 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.100215    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:44:55.100301 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.100240    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-96.ec2.internal"
Sep 11 17:44:55.110331 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.110309    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-96.ec2.internal"
Sep 11 17:44:55.110411 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:55.110333    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-96.ec2.internal\": node \"ip-10-0-138-96.ec2.internal\" not found"
Sep 11 17:44:55.138802 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:55.138641    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-96.ec2.internal\" not found"
Sep 11 17:44:55.147116 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.147095    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-96.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-96.ec2.internal"]
Sep 11 17:44:55.147185 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.147159    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:44:55.148036 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.148021    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:44:55.148116 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.148047    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:44:55.148116 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.148058    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:44:55.150381 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.150366    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:44:55.150489 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.150474    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-96.ec2.internal"
Sep 11 17:44:55.150552 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.150526    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:44:55.151149 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.151132    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:44:55.151149 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.151143    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:44:55.151263 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.151163    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:44:55.151263 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.151164    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:44:55.151263 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.151174    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:44:55.151263 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.151182    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:44:55.153311 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.153292    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-96.ec2.internal"
Sep 11 17:44:55.153434 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.153322    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:44:55.154010 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.153979    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:44:55.154091 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.154023    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:44:55.154091 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.154034    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:44:55.168877 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:55.168853    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-96.ec2.internal\" not found" node="ip-10-0-138-96.ec2.internal"
Sep 11 17:44:55.173085 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:55.173069    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-96.ec2.internal\" not found" node="ip-10-0-138-96.ec2.internal"
Sep 11 17:44:55.239385 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:55.239340    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-96.ec2.internal\" not found"
Sep 11 17:44:55.244208 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.244189    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aa3822d42868560dd3e2a5d526ebe096-config\") pod \"kube-apiserver-proxy-ip-10-0-138-96.ec2.internal\" (UID: \"aa3822d42868560dd3e2a5d526ebe096\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-96.ec2.internal"
Sep 11 17:44:55.244272 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.244220    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6194dcd4cf42d1c4c056092b6c620840-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-96.ec2.internal\" (UID: \"6194dcd4cf42d1c4c056092b6c620840\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-96.ec2.internal"
Sep 11 17:44:55.244272 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.244260    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6194dcd4cf42d1c4c056092b6c620840-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-96.ec2.internal\" (UID: \"6194dcd4cf42d1c4c056092b6c620840\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-96.ec2.internal"
Sep 11 17:44:55.340444 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:55.340417    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-96.ec2.internal\" not found"
Sep 11 17:44:55.344778 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.344763    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6194dcd4cf42d1c4c056092b6c620840-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-96.ec2.internal\" (UID: \"6194dcd4cf42d1c4c056092b6c620840\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-96.ec2.internal"
Sep 11 17:44:55.344835 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.344786    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6194dcd4cf42d1c4c056092b6c620840-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-96.ec2.internal\" (UID: \"6194dcd4cf42d1c4c056092b6c620840\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-96.ec2.internal"
Sep 11 17:44:55.344835 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.344826    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aa3822d42868560dd3e2a5d526ebe096-config\") pod \"kube-apiserver-proxy-ip-10-0-138-96.ec2.internal\" (UID: \"aa3822d42868560dd3e2a5d526ebe096\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-96.ec2.internal"
Sep 11 17:44:55.344912 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.344852    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aa3822d42868560dd3e2a5d526ebe096-config\") pod \"kube-apiserver-proxy-ip-10-0-138-96.ec2.internal\" (UID: \"aa3822d42868560dd3e2a5d526ebe096\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-96.ec2.internal"
Sep 11 17:44:55.344912 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.344861    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6194dcd4cf42d1c4c056092b6c620840-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-96.ec2.internal\" (UID: \"6194dcd4cf42d1c4c056092b6c620840\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-96.ec2.internal"
Sep 11 17:44:55.344912 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.344858    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6194dcd4cf42d1c4c056092b6c620840-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-96.ec2.internal\" (UID: \"6194dcd4cf42d1c4c056092b6c620840\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-96.ec2.internal"
Sep 11 17:44:55.441436 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:55.441397    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-96.ec2.internal\" not found"
Sep 11 17:44:55.473190 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.473155    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-96.ec2.internal"
Sep 11 17:44:55.476504 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.476478    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-96.ec2.internal"
Sep 11 17:44:55.542454 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:55.542413    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-96.ec2.internal\" not found"
Sep 11 17:44:55.647436 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:55.647386    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-96.ec2.internal\" not found"
Sep 11 17:44:55.748626 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:55.748559    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-96.ec2.internal\" not found"
Sep 11 17:44:55.848914 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:55.848881    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-96.ec2.internal\" not found"
Sep 11 17:44:55.866631 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.866598    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 17:44:55.866781 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.866758    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 17:44:55.949772 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:55.949746    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-96.ec2.internal\" not found"
Sep 11 17:44:55.975252 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.975211    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 17:39:55 +0000 UTC" deadline="2028-04-04 13:00:02.208635526 +0000 UTC"
Sep 11 17:44:55.975252 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:55.975248    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13699h15m6.23339135s"
Sep 11 17:44:56.050468 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:56.050440    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-96.ec2.internal\" not found"
Sep 11 17:44:56.103689 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.103654    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:44:56.105050 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.105025    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:44:56.105172 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.105070    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:44:56.105172 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.105083    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:44:56.150912 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:56.150874    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-96.ec2.internal\" not found"
Sep 11 17:44:56.182454 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:56.182407    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6194dcd4cf42d1c4c056092b6c620840.slice/crio-134da1a025c3faff8f8e15ce313948c8f038797e32e48cc5c6b5698237da5305 WatchSource:0}: Error finding container 134da1a025c3faff8f8e15ce313948c8f038797e32e48cc5c6b5698237da5305: Status 404 returned error can't find the container with id 134da1a025c3faff8f8e15ce313948c8f038797e32e48cc5c6b5698237da5305
Sep 11 17:44:56.183033 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:56.183008    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa3822d42868560dd3e2a5d526ebe096.slice/crio-7c2aa98a403b54e73ff22068d8bbcac922566a745d19b08c4d17b8a34a67fe2d WatchSource:0}: Error finding container 7c2aa98a403b54e73ff22068d8bbcac922566a745d19b08c4d17b8a34a67fe2d: Status 404 returned error can't find the container with id 7c2aa98a403b54e73ff22068d8bbcac922566a745d19b08c4d17b8a34a67fe2d
Sep 11 17:44:56.186985 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.186971    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:44:56.251673 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:56.251631    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-96.ec2.internal\" not found"
Sep 11 17:44:56.302717 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.302633    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:44:56.344460 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.344421    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-96.ec2.internal"
Sep 11 17:44:56.362361 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.362333    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 17:44:56.364567 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.364550    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-96.ec2.internal"
Sep 11 17:44:56.374127 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.374104    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 17:44:56.428538 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.428511    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:44:56.505496 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.505462    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:44:56.709284 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.709199    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:44:56.918829 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.918784    2559 apiserver.go:52] "Watching apiserver"
Sep 11 17:44:56.924696 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.924667    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 17:44:56.928148 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.928106    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-hzjgd","kube-system/konnectivity-agent-p2znc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6","openshift-cluster-node-tuning-operator/tuned-lqlkq","openshift-dns/node-resolver-5qftl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-96.ec2.internal","openshift-multus/multus-additional-cni-plugins-d9qt2","openshift-multus/network-metrics-daemon-wb9dr","openshift-network-diagnostics/network-check-target-8nchm","kube-system/global-pull-secret-syncer-rsgml","kube-system/kube-apiserver-proxy-ip-10-0-138-96.ec2.internal","openshift-image-registry/node-ca-kpjjv","openshift-multus/multus-wh2s7","openshift-network-operator/iptables-alerter-pjtfk"]
Sep 11 17:44:56.931276 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.931253    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:44:56.931422 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:56.931342    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb9dr" podUID="5ee7b9a9-a331-4b43-b68e-439edfc2dfd6"
Sep 11 17:44:56.933480 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.933451    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p2znc"
Sep 11 17:44:56.935873 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.935836    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6"
Sep 11 17:44:56.937134 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.937114    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-k8r4w\""
Sep 11 17:44:56.937256 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.937240    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 17:44:56.937535 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.937515    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 17:44:56.938260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.938235    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:56.938425 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.938408    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 17:44:56.938698 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.938679    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 17:44:56.939045 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.939030    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 17:44:56.939130 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.939025    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jdmvx\""
Sep 11 17:44:56.940652 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.940617    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 17:44:56.940856 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.940836    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jj85b\""
Sep 11 17:44:56.940967 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.940943    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5qftl"
Sep 11 17:44:56.941120 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.941100    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 17:44:56.941120 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.941118    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 17:44:56.941317 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.940948    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 17:44:56.942201 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.942179    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 17:44:56.944191 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.943233    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 17:44:56.946606 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.946413    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:44:56.946606 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:56.946527    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nchm" podUID="f7aa7737-b530-483f-b132-4768683949d3"
Sep 11 17:44:56.949137 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.946758    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 17:44:56.949137 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.946830    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f6kk2\""
Sep 11 17:44:56.950532 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.950510    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d9qt2"
Sep 11 17:44:56.952970 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.952947    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 17:44:56.953337 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.953105    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 17:44:56.953337 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.953226    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8bgjb\""
Sep 11 17:44:56.954567 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.954541    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-multus-socket-dir-parent\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:56.954668 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.954583    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-hostroot\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:56.954668 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.954613    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-host-run-multus-certs\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:56.954668 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.954642    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/190c54c5-ae20-4ef1-b9b2-af966c1712f4-hosts-file\") pod \"node-resolver-5qftl\" (UID: \"190c54c5-ae20-4ef1-b9b2-af966c1712f4\") " pod="openshift-dns/node-resolver-5qftl"
Sep 11 17:44:56.954835 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.954668    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c14450ba-245b-4010-9ece-ee27d149274a-socket-dir\") pod \"aws-ebs-csi-driver-node-fkqb6\" (UID: \"c14450ba-245b-4010-9ece-ee27d149274a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6"
Sep 11 17:44:56.954835 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.954692    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db5xf\" (UniqueName: \"kubernetes.io/projected/c14450ba-245b-4010-9ece-ee27d149274a-kube-api-access-db5xf\") pod \"aws-ebs-csi-driver-node-fkqb6\" (UID: \"c14450ba-245b-4010-9ece-ee27d149274a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6"
Sep 11 17:44:56.954835 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.954716    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-cnibin\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:56.954835 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.954740    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190c54c5-ae20-4ef1-b9b2-af966c1712f4-tmp-dir\") pod \"node-resolver-5qftl\" (UID: \"190c54c5-ae20-4ef1-b9b2-af966c1712f4\") " pod="openshift-dns/node-resolver-5qftl"
Sep 11 17:44:56.954835 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.954765    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw6c9\" (UniqueName: \"kubernetes.io/projected/f7aa7737-b530-483f-b132-4768683949d3-kube-api-access-lw6c9\") pod \"network-check-target-8nchm\" (UID: \"f7aa7737-b530-483f-b132-4768683949d3\") " pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:44:56.954835 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.954828    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-cni-binary-copy\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:56.955152 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.954856    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-host-var-lib-cni-multus\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:56.955152 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.954887    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-host-var-lib-kubelet\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:56.955152 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.954913    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c14450ba-245b-4010-9ece-ee27d149274a-etc-selinux\") pod \"aws-ebs-csi-driver-node-fkqb6\" (UID: \"c14450ba-245b-4010-9ece-ee27d149274a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6"
Sep 11 17:44:56.955152 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.954949    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-multus-cni-dir\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:56.955152 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.954952    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:56.955493 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.954973    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-host-var-lib-cni-bin\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:56.955493 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.955217    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-etc-kubernetes\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:56.955493 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.955243    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-system-cni-dir\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:56.955493 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.955267    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-multus-conf-dir\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:56.955493 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.955293    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zstrq\" (UniqueName: \"kubernetes.io/projected/190c54c5-ae20-4ef1-b9b2-af966c1712f4-kube-api-access-zstrq\") pod \"node-resolver-5qftl\" (UID: \"190c54c5-ae20-4ef1-b9b2-af966c1712f4\") " pod="openshift-dns/node-resolver-5qftl"
Sep 11 17:44:56.955493 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.955327    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a65223d8-9c38-454a-a958-120affb54b81-agent-certs\") pod \"konnectivity-agent-p2znc\" (UID: \"a65223d8-9c38-454a-a958-120affb54b81\") " pod="kube-system/konnectivity-agent-p2znc"
Sep 11 17:44:56.955493 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.955353    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c14450ba-245b-4010-9ece-ee27d149274a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fkqb6\" (UID: \"c14450ba-245b-4010-9ece-ee27d149274a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6"
Sep 11 17:44:56.955493 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.955400    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-host-run-k8s-cni-cncf-io\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:56.955493 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.955425    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-host-run-netns\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:56.955493 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.955450    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs\") pod \"network-metrics-daemon-wb9dr\" (UID: \"5ee7b9a9-a331-4b43-b68e-439edfc2dfd6\") " pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:44:56.955493 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.955473    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f7xk\" (UniqueName: \"kubernetes.io/projected/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-kube-api-access-7f7xk\") pod \"network-metrics-daemon-wb9dr\" (UID: \"5ee7b9a9-a331-4b43-b68e-439edfc2dfd6\") " pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:44:56.956079 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.955530    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c14450ba-245b-4010-9ece-ee27d149274a-device-dir\") pod \"aws-ebs-csi-driver-node-fkqb6\" (UID: \"c14450ba-245b-4010-9ece-ee27d149274a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6"
Sep 11 17:44:56.956079 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.955581    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-multus-daemon-config\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:56.956079 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.955629    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5mn5\" (UniqueName: \"kubernetes.io/projected/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-kube-api-access-z5mn5\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:56.956079 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.955697    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a65223d8-9c38-454a-a958-120affb54b81-konnectivity-ca\") pod \"konnectivity-agent-p2znc\" (UID: \"a65223d8-9c38-454a-a958-120affb54b81\") " pod="kube-system/konnectivity-agent-p2znc"
Sep 11 17:44:56.956079 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.955760    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c14450ba-245b-4010-9ece-ee27d149274a-registration-dir\") pod \"aws-ebs-csi-driver-node-fkqb6\" (UID: \"c14450ba-245b-4010-9ece-ee27d149274a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6"
Sep 11 17:44:56.956079 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.955787    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c14450ba-245b-4010-9ece-ee27d149274a-sys-fs\") pod \"aws-ebs-csi-driver-node-fkqb6\" (UID: \"c14450ba-245b-4010-9ece-ee27d149274a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6"
Sep 11 17:44:56.956079 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.955823    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 17:44:56.957386 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.957107    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6q9p6\""
Sep 11 17:44:56.957386 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.957334    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:44:56.958145 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.957711    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 17:44:56.958145 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.957774    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:44:56.958145 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:56.957855    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsgml" podUID="8890e4f4-b248-4fe7-b3e7-96fec145ce34"
Sep 11 17:44:56.960600 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.960539    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kpjjv"
Sep 11 17:44:56.963795 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.963775    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:56.964667 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.964643    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 17:44:56.964773 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.964704    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 17:44:56.964840 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.964826    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jtlwp\""
Sep 11 17:44:56.964898 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.964647    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 17:44:56.966427 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.966053    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pjtfk"
Sep 11 17:44:56.967906 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.966654    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 17:44:56.967906 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.967079    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 17:44:56.967906 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.967308    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 17:44:56.967906 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.967537    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8xgtg\""
Sep 11 17:44:56.967906 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.967678    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 17:44:56.970267 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.968069    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 17:44:56.970267 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.968300    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 17:44:56.970267 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.968703    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:44:56.972355 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.970872    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 17:44:56.972355 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.971576    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-s2bzh\""
Sep 11 17:44:56.972355 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.971779    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 17:44:56.981564 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.981543    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-86bbk"
Sep 11 17:44:56.994812 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:56.994554    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-86bbk"
Sep 11 17:44:57.044359 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.044272    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 17:44:57.052294 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.052246    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-96.ec2.internal" event={"ID":"6194dcd4cf42d1c4c056092b6c620840","Type":"ContainerStarted","Data":"134da1a025c3faff8f8e15ce313948c8f038797e32e48cc5c6b5698237da5305"}
Sep 11 17:44:57.053896 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.053865    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-96.ec2.internal" event={"ID":"aa3822d42868560dd3e2a5d526ebe096","Type":"ContainerStarted","Data":"7c2aa98a403b54e73ff22068d8bbcac922566a745d19b08c4d17b8a34a67fe2d"}
Sep 11 17:44:57.056072 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056044    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-host-run-netns\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.056186 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056086    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-etc-openvswitch\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.056186 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056117    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c14450ba-245b-4010-9ece-ee27d149274a-device-dir\") pod \"aws-ebs-csi-driver-node-fkqb6\" (UID: \"c14450ba-245b-4010-9ece-ee27d149274a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6"
Sep 11 17:44:57.056186 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056145    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5mn5\" (UniqueName: \"kubernetes.io/projected/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-kube-api-access-z5mn5\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.056348 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056220    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-node-log\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.056348 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056249    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bxrk\" (UniqueName: \"kubernetes.io/projected/c8bfa2da-0418-44c2-801b-e5b44882a17b-kube-api-access-8bxrk\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.056348 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056276    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-etc-systemd\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.056348 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056294    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/500ad4eb-717e-4ab2-879e-ea321c6259f3-etc-tuned\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.056348 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056314    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a65223d8-9c38-454a-a958-120affb54b81-konnectivity-ca\") pod \"konnectivity-agent-p2znc\" (UID: \"a65223d8-9c38-454a-a958-120affb54b81\") " pod="kube-system/konnectivity-agent-p2znc"
Sep 11 17:44:57.056348 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056333    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c14450ba-245b-4010-9ece-ee27d149274a-registration-dir\") pod \"aws-ebs-csi-driver-node-fkqb6\" (UID: \"c14450ba-245b-4010-9ece-ee27d149274a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6"
Sep 11 17:44:57.056672 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056226    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c14450ba-245b-4010-9ece-ee27d149274a-device-dir\") pod \"aws-ebs-csi-driver-node-fkqb6\" (UID: \"c14450ba-245b-4010-9ece-ee27d149274a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6"
Sep 11 17:44:57.056672 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056351    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c14450ba-245b-4010-9ece-ee27d149274a-sys-fs\") pod \"aws-ebs-csi-driver-node-fkqb6\" (UID: \"c14450ba-245b-4010-9ece-ee27d149274a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6"
Sep 11 17:44:57.056672 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056417    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c14450ba-245b-4010-9ece-ee27d149274a-sys-fs\") pod \"aws-ebs-csi-driver-node-fkqb6\" (UID: \"c14450ba-245b-4010-9ece-ee27d149274a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6"
Sep 11 17:44:57.056672 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056406    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-multus-socket-dir-parent\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.056672 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056463    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21f61290-cf4a-4d5b-a1fc-694a52fde254-host-slash\") pod \"iptables-alerter-pjtfk\" (UID: \"21f61290-cf4a-4d5b-a1fc-694a52fde254\") " pod="openshift-network-operator/iptables-alerter-pjtfk"
Sep 11 17:44:57.056672 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056491    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-run\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.056672 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056521    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp8bd\" (UniqueName: \"kubernetes.io/projected/500ad4eb-717e-4ab2-879e-ea321c6259f3-kube-api-access-jp8bd\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.056672 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056554    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db5xf\" (UniqueName: \"kubernetes.io/projected/c14450ba-245b-4010-9ece-ee27d149274a-kube-api-access-db5xf\") pod \"aws-ebs-csi-driver-node-fkqb6\" (UID: \"c14450ba-245b-4010-9ece-ee27d149274a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6"
Sep 11 17:44:57.056672 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056582    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-d9qt2\" (UID: \"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8\") " pod="openshift-multus/multus-additional-cni-plugins-d9qt2"
Sep 11 17:44:57.056672 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056611    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d9qt2\" (UID: \"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8\") " pod="openshift-multus/multus-additional-cni-plugins-d9qt2"
Sep 11 17:44:57.056672 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056636    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8238ce9-9f28-4900-8b41-667fa71978f2-host\") pod \"node-ca-kpjjv\" (UID: \"a8238ce9-9f28-4900-8b41-667fa71978f2\") " pod="openshift-image-registry/node-ca-kpjjv"
Sep 11 17:44:57.056672 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056662    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-run-ovn\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.056672 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056666    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-multus-socket-dir-parent\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.057321 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056687    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8bfa2da-0418-44c2-801b-e5b44882a17b-ovn-node-metrics-cert\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.057321 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056714    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-host-var-lib-kubelet\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.057321 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056739    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-lib-modules\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.057321 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056765    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-multus-cni-dir\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.057321 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056789    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c8bfa2da-0418-44c2-801b-e5b44882a17b-ovnkube-script-lib\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.057321 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056815    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-etc-kubernetes\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.057321 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056839    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8-system-cni-dir\") pod \"multus-additional-cni-plugins-d9qt2\" (UID: \"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8\") " pod="openshift-multus/multus-additional-cni-plugins-d9qt2"
Sep 11 17:44:57.057321 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056866    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-host-run-ovn-kubernetes\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.057321 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056890    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c8bfa2da-0418-44c2-801b-e5b44882a17b-ovnkube-config\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.057321 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056916    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-system-cni-dir\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.057321 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056915    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a65223d8-9c38-454a-a958-120affb54b81-konnectivity-ca\") pod \"konnectivity-agent-p2znc\" (UID: \"a65223d8-9c38-454a-a958-120affb54b81\") " pod="kube-system/konnectivity-agent-p2znc"
Sep 11 17:44:57.057321 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.056946    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-multus-conf-dir\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.057321 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057012    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zstrq\" (UniqueName: \"kubernetes.io/projected/190c54c5-ae20-4ef1-b9b2-af966c1712f4-kube-api-access-zstrq\") pod \"node-resolver-5qftl\" (UID: \"190c54c5-ae20-4ef1-b9b2-af966c1712f4\") " pod="openshift-dns/node-resolver-5qftl"
Sep 11 17:44:57.057321 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057018    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-system-cni-dir\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.057321 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057045    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8890e4f4-b248-4fe7-b3e7-96fec145ce34-dbus\") pod \"global-pull-secret-syncer-rsgml\" (UID: \"8890e4f4-b248-4fe7-b3e7-96fec145ce34\") " pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:44:57.057321 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057018    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-etc-kubernetes\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.057321 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057072    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-host-var-lib-kubelet\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.058260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057073    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8-cnibin\") pod \"multus-additional-cni-plugins-d9qt2\" (UID: \"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8\") " pod="openshift-multus/multus-additional-cni-plugins-d9qt2"
Sep 11 17:44:57.058260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057090    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c14450ba-245b-4010-9ece-ee27d149274a-registration-dir\") pod \"aws-ebs-csi-driver-node-fkqb6\" (UID: \"c14450ba-245b-4010-9ece-ee27d149274a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6"
Sep 11 17:44:57.058260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057120    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-host-kubelet\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.058260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057130    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-multus-conf-dir\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.058260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057188    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-run-systemd\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.058260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057217    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-run-openvswitch\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.058260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057249    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c14450ba-245b-4010-9ece-ee27d149274a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fkqb6\" (UID: \"c14450ba-245b-4010-9ece-ee27d149274a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6"
Sep 11 17:44:57.058260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057257    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-multus-cni-dir\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.058260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057287    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/21f61290-cf4a-4d5b-a1fc-694a52fde254-iptables-alerter-script\") pod \"iptables-alerter-pjtfk\" (UID: \"21f61290-cf4a-4d5b-a1fc-694a52fde254\") " pod="openshift-network-operator/iptables-alerter-pjtfk"
Sep 11 17:44:57.058260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057328    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-etc-modprobe-d\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.058260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057361    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-host\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.058260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057375    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c14450ba-245b-4010-9ece-ee27d149274a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fkqb6\" (UID: \"c14450ba-245b-4010-9ece-ee27d149274a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6"
Sep 11 17:44:57.058260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057404    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-multus-daemon-config\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.058260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057462    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v48xf\" (UniqueName: \"kubernetes.io/projected/a8238ce9-9f28-4900-8b41-667fa71978f2-kube-api-access-v48xf\") pod \"node-ca-kpjjv\" (UID: \"a8238ce9-9f28-4900-8b41-667fa71978f2\") " pod="openshift-image-registry/node-ca-kpjjv"
Sep 11 17:44:57.058260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057551    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190c54c5-ae20-4ef1-b9b2-af966c1712f4-tmp-dir\") pod \"node-resolver-5qftl\" (UID: \"190c54c5-ae20-4ef1-b9b2-af966c1712f4\") " pod="openshift-dns/node-resolver-5qftl"
Sep 11 17:44:57.058260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057589    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw6c9\" (UniqueName: \"kubernetes.io/projected/f7aa7737-b530-483f-b132-4768683949d3-kube-api-access-lw6c9\") pod \"network-check-target-8nchm\" (UID: \"f7aa7737-b530-483f-b132-4768683949d3\") " pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:44:57.058260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057665    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8890e4f4-b248-4fe7-b3e7-96fec145ce34-kubelet-config\") pod \"global-pull-secret-syncer-rsgml\" (UID: \"8890e4f4-b248-4fe7-b3e7-96fec145ce34\") " pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:44:57.059235 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057696    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a8238ce9-9f28-4900-8b41-667fa71978f2-serviceca\") pod \"node-ca-kpjjv\" (UID: \"a8238ce9-9f28-4900-8b41-667fa71978f2\") " pod="openshift-image-registry/node-ca-kpjjv"
Sep 11 17:44:57.059235 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057721    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-var-lib-openvswitch\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.059235 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057748    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-hostroot\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.059235 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057802    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-host-run-multus-certs\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.059235 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057827    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/190c54c5-ae20-4ef1-b9b2-af966c1712f4-hosts-file\") pod \"node-resolver-5qftl\" (UID: \"190c54c5-ae20-4ef1-b9b2-af966c1712f4\") " pod="openshift-dns/node-resolver-5qftl"
Sep 11 17:44:57.059235 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057855    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c14450ba-245b-4010-9ece-ee27d149274a-socket-dir\") pod \"aws-ebs-csi-driver-node-fkqb6\" (UID: \"c14450ba-245b-4010-9ece-ee27d149274a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6"
Sep 11 17:44:57.059235 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057881    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-cnibin\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.059235 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057907    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8890e4f4-b248-4fe7-b3e7-96fec145ce34-original-pull-secret\") pod \"global-pull-secret-syncer-rsgml\" (UID: \"8890e4f4-b248-4fe7-b3e7-96fec145ce34\") " pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:44:57.059235 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057932    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-host-slash\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.059235 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057957    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-host-cni-netd\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.059235 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.057985    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.059235 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058012    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-multus-daemon-config\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.059235 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058069    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xntl6\" (UniqueName: \"kubernetes.io/projected/21f61290-cf4a-4d5b-a1fc-694a52fde254-kube-api-access-xntl6\") pod \"iptables-alerter-pjtfk\" (UID: \"21f61290-cf4a-4d5b-a1fc-694a52fde254\") " pod="openshift-network-operator/iptables-alerter-pjtfk"
Sep 11 17:44:57.059235 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058101    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-etc-kubernetes\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.059235 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058134    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-cni-binary-copy\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.059235 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058164    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-host-var-lib-cni-multus\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.059235 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058196    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d9qt2\" (UID: \"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8\") " pod="openshift-multus/multus-additional-cni-plugins-d9qt2"
Sep 11 17:44:57.060232 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058225    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-etc-sysconfig\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.060232 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058251    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-etc-sysctl-d\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.060232 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058279    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c14450ba-245b-4010-9ece-ee27d149274a-etc-selinux\") pod \"aws-ebs-csi-driver-node-fkqb6\" (UID: \"c14450ba-245b-4010-9ece-ee27d149274a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6"
Sep 11 17:44:57.060232 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058305    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-host-var-lib-cni-bin\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.060232 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058335    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d9qt2\" (UID: \"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8\") " pod="openshift-multus/multus-additional-cni-plugins-d9qt2"
Sep 11 17:44:57.060232 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058363    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz2ql\" (UniqueName: \"kubernetes.io/projected/d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8-kube-api-access-mz2ql\") pod \"multus-additional-cni-plugins-d9qt2\" (UID: \"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8\") " pod="openshift-multus/multus-additional-cni-plugins-d9qt2"
Sep 11 17:44:57.060232 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058074    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-hostroot\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.060232 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058504    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c8bfa2da-0418-44c2-801b-e5b44882a17b-env-overrides\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.060232 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058534    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-etc-sysctl-conf\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.060232 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058561    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-sys\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.060232 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058565    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190c54c5-ae20-4ef1-b9b2-af966c1712f4-tmp-dir\") pod \"node-resolver-5qftl\" (UID: \"190c54c5-ae20-4ef1-b9b2-af966c1712f4\") " pod="openshift-dns/node-resolver-5qftl"
Sep 11 17:44:57.060232 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058587    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/500ad4eb-717e-4ab2-879e-ea321c6259f3-tmp\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.060232 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058618    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-systemd-units\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.060232 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058641    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-host-run-multus-certs\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.060232 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058644    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-log-socket\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.060232 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058683    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-host-cni-bin\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.060232 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058712    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-var-lib-kubelet\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.061155 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058743    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a65223d8-9c38-454a-a958-120affb54b81-agent-certs\") pod \"konnectivity-agent-p2znc\" (UID: \"a65223d8-9c38-454a-a958-120affb54b81\") " pod="kube-system/konnectivity-agent-p2znc"
Sep 11 17:44:57.061155 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058782    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-host-run-k8s-cni-cncf-io\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.061155 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058812    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-host-run-netns\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.061155 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058839    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs\") pod \"network-metrics-daemon-wb9dr\" (UID: \"5ee7b9a9-a331-4b43-b68e-439edfc2dfd6\") " pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:44:57.061155 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058866    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f7xk\" (UniqueName: \"kubernetes.io/projected/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-kube-api-access-7f7xk\") pod \"network-metrics-daemon-wb9dr\" (UID: \"5ee7b9a9-a331-4b43-b68e-439edfc2dfd6\") " pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:44:57.061155 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.058966    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/190c54c5-ae20-4ef1-b9b2-af966c1712f4-hosts-file\") pod \"node-resolver-5qftl\" (UID: \"190c54c5-ae20-4ef1-b9b2-af966c1712f4\") " pod="openshift-dns/node-resolver-5qftl"
Sep 11 17:44:57.061155 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.059059    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c14450ba-245b-4010-9ece-ee27d149274a-socket-dir\") pod \"aws-ebs-csi-driver-node-fkqb6\" (UID: \"c14450ba-245b-4010-9ece-ee27d149274a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6"
Sep 11 17:44:57.061155 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.059102    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-cnibin\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.061155 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.059118    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-cni-binary-copy\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.061155 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.059173    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-host-var-lib-cni-multus\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.061155 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.059176    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-host-run-k8s-cni-cncf-io\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.061155 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.059227    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-host-var-lib-cni-bin\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.061155 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.059497    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 17:44:57.061155 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:57.059531    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:44:57.061155 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.059579    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-host-run-netns\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.061155 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:57.059679    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs podName:5ee7b9a9-a331-4b43-b68e-439edfc2dfd6 nodeName:}" failed. No retries permitted until 2026-09-11 17:44:57.559639456 +0000 UTC m=+3.004740029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs") pod "network-metrics-daemon-wb9dr" (UID: "5ee7b9a9-a331-4b43-b68e-439edfc2dfd6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:44:57.061155 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.059783    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c14450ba-245b-4010-9ece-ee27d149274a-etc-selinux\") pod \"aws-ebs-csi-driver-node-fkqb6\" (UID: \"c14450ba-245b-4010-9ece-ee27d149274a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6"
Sep 11 17:44:57.065622 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:57.065414    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:44:57.065622 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:57.065454    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:44:57.065622 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:57.065478    2559 projected.go:196] Error preparing data for projected volume kube-api-access-lw6c9 for pod openshift-network-diagnostics/network-check-target-8nchm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:44:57.065622 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:57.065536    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7aa7737-b530-483f-b132-4768683949d3-kube-api-access-lw6c9 podName:f7aa7737-b530-483f-b132-4768683949d3 nodeName:}" failed. No retries permitted until 2026-09-11 17:44:57.565517227 +0000 UTC m=+3.010617778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lw6c9" (UniqueName: "kubernetes.io/projected/f7aa7737-b530-483f-b132-4768683949d3-kube-api-access-lw6c9") pod "network-check-target-8nchm" (UID: "f7aa7737-b530-483f-b132-4768683949d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:44:57.067112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.067086    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5mn5\" (UniqueName: \"kubernetes.io/projected/7d33fe75-e2cd-43e9-9ab1-0078a32c48cc-kube-api-access-z5mn5\") pod \"multus-wh2s7\" (UID: \"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc\") " pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.067240 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.067099    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zstrq\" (UniqueName: \"kubernetes.io/projected/190c54c5-ae20-4ef1-b9b2-af966c1712f4-kube-api-access-zstrq\") pod \"node-resolver-5qftl\" (UID: \"190c54c5-ae20-4ef1-b9b2-af966c1712f4\") " pod="openshift-dns/node-resolver-5qftl"
Sep 11 17:44:57.068139 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.068115    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a65223d8-9c38-454a-a958-120affb54b81-agent-certs\") pod \"konnectivity-agent-p2znc\" (UID: \"a65223d8-9c38-454a-a958-120affb54b81\") " pod="kube-system/konnectivity-agent-p2znc"
Sep 11 17:44:57.070152 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.070127    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db5xf\" (UniqueName: \"kubernetes.io/projected/c14450ba-245b-4010-9ece-ee27d149274a-kube-api-access-db5xf\") pod \"aws-ebs-csi-driver-node-fkqb6\" (UID: \"c14450ba-245b-4010-9ece-ee27d149274a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6"
Sep 11 17:44:57.072174 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.072106    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f7xk\" (UniqueName: \"kubernetes.io/projected/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-kube-api-access-7f7xk\") pod \"network-metrics-daemon-wb9dr\" (UID: \"5ee7b9a9-a331-4b43-b68e-439edfc2dfd6\") " pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:44:57.159675 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.159635    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-node-log\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.159877 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.159685    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bxrk\" (UniqueName: \"kubernetes.io/projected/c8bfa2da-0418-44c2-801b-e5b44882a17b-kube-api-access-8bxrk\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.159877 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.159764    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-node-log\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.159877 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.159846    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-etc-systemd\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.160053 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.159881    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/500ad4eb-717e-4ab2-879e-ea321c6259f3-etc-tuned\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.160053 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.159916    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21f61290-cf4a-4d5b-a1fc-694a52fde254-host-slash\") pod \"iptables-alerter-pjtfk\" (UID: \"21f61290-cf4a-4d5b-a1fc-694a52fde254\") " pod="openshift-network-operator/iptables-alerter-pjtfk"
Sep 11 17:44:57.160053 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.159931    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-etc-systemd\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.160053 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.159942    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-run\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.160053 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.159968    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp8bd\" (UniqueName: \"kubernetes.io/projected/500ad4eb-717e-4ab2-879e-ea321c6259f3-kube-api-access-jp8bd\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.160053 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.159985    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21f61290-cf4a-4d5b-a1fc-694a52fde254-host-slash\") pod \"iptables-alerter-pjtfk\" (UID: \"21f61290-cf4a-4d5b-a1fc-694a52fde254\") " pod="openshift-network-operator/iptables-alerter-pjtfk"
Sep 11 17:44:57.160053 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.159999    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-d9qt2\" (UID: \"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8\") " pod="openshift-multus/multus-additional-cni-plugins-d9qt2"
Sep 11 17:44:57.160053 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160025    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d9qt2\" (UID: \"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8\") " pod="openshift-multus/multus-additional-cni-plugins-d9qt2"
Sep 11 17:44:57.160053 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160050    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8238ce9-9f28-4900-8b41-667fa71978f2-host\") pod \"node-ca-kpjjv\" (UID: \"a8238ce9-9f28-4900-8b41-667fa71978f2\") " pod="openshift-image-registry/node-ca-kpjjv"
Sep 11 17:44:57.160495 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160037    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-run\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.160495 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160077    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-run-ovn\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.160495 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160102    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8bfa2da-0418-44c2-801b-e5b44882a17b-ovn-node-metrics-cert\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.160495 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160126    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-lib-modules\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.160495 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160151    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c8bfa2da-0418-44c2-801b-e5b44882a17b-ovnkube-script-lib\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.160495 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160176    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8-system-cni-dir\") pod \"multus-additional-cni-plugins-d9qt2\" (UID: \"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8\") " pod="openshift-multus/multus-additional-cni-plugins-d9qt2"
Sep 11 17:44:57.160495 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160201    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-host-run-ovn-kubernetes\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.160495 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160226    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c8bfa2da-0418-44c2-801b-e5b44882a17b-ovnkube-config\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.160495 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160255    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8890e4f4-b248-4fe7-b3e7-96fec145ce34-dbus\") pod \"global-pull-secret-syncer-rsgml\" (UID: \"8890e4f4-b248-4fe7-b3e7-96fec145ce34\") " pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:44:57.160495 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160286    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8-cnibin\") pod \"multus-additional-cni-plugins-d9qt2\" (UID: \"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8\") " pod="openshift-multus/multus-additional-cni-plugins-d9qt2"
Sep 11 17:44:57.160495 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160365    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8238ce9-9f28-4900-8b41-667fa71978f2-host\") pod \"node-ca-kpjjv\" (UID: \"a8238ce9-9f28-4900-8b41-667fa71978f2\") " pod="openshift-image-registry/node-ca-kpjjv"
Sep 11 17:44:57.160495 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160439    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8-cnibin\") pod \"multus-additional-cni-plugins-d9qt2\" (UID: \"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8\") " pod="openshift-multus/multus-additional-cni-plugins-d9qt2"
Sep 11 17:44:57.160495 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160440    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-lib-modules\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.160495 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160470    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-run-ovn\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.161260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160519    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-host-kubelet\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.161260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160547    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-run-systemd\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.161260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160577    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-run-openvswitch\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.161260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160589    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-host-run-ovn-kubernetes\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.161260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160606    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/21f61290-cf4a-4d5b-a1fc-694a52fde254-iptables-alerter-script\") pod \"iptables-alerter-pjtfk\" (UID: \"21f61290-cf4a-4d5b-a1fc-694a52fde254\") " pod="openshift-network-operator/iptables-alerter-pjtfk"
Sep 11 17:44:57.161260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160602    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d9qt2\" (UID: \"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8\") " pod="openshift-multus/multus-additional-cni-plugins-d9qt2"
Sep 11 17:44:57.161260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160647    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-etc-modprobe-d\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.161260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160654    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-host-kubelet\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.161260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160727    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-run-systemd\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.161260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160766    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8-system-cni-dir\") pod \"multus-additional-cni-plugins-d9qt2\" (UID: \"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8\") " pod="openshift-multus/multus-additional-cni-plugins-d9qt2"
Sep 11 17:44:57.161260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160769    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-etc-modprobe-d\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.161260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160781    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8890e4f4-b248-4fe7-b3e7-96fec145ce34-dbus\") pod \"global-pull-secret-syncer-rsgml\" (UID: \"8890e4f4-b248-4fe7-b3e7-96fec145ce34\") " pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:44:57.161260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160817    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-host\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.161260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160851    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v48xf\" (UniqueName: \"kubernetes.io/projected/a8238ce9-9f28-4900-8b41-667fa71978f2-kube-api-access-v48xf\") pod \"node-ca-kpjjv\" (UID: \"a8238ce9-9f28-4900-8b41-667fa71978f2\") " pod="openshift-image-registry/node-ca-kpjjv"
Sep 11 17:44:57.161260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160896    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8890e4f4-b248-4fe7-b3e7-96fec145ce34-kubelet-config\") pod \"global-pull-secret-syncer-rsgml\" (UID: \"8890e4f4-b248-4fe7-b3e7-96fec145ce34\") " pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:44:57.161260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160914    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c8bfa2da-0418-44c2-801b-e5b44882a17b-ovnkube-script-lib\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.161260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160922    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a8238ce9-9f28-4900-8b41-667fa71978f2-serviceca\") pod \"node-ca-kpjjv\" (UID: \"a8238ce9-9f28-4900-8b41-667fa71978f2\") " pod="openshift-image-registry/node-ca-kpjjv"
Sep 11 17:44:57.162143 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160950    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-var-lib-openvswitch\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.162143 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.160984    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8890e4f4-b248-4fe7-b3e7-96fec145ce34-original-pull-secret\") pod \"global-pull-secret-syncer-rsgml\" (UID: \"8890e4f4-b248-4fe7-b3e7-96fec145ce34\") " pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:44:57.162143 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161012    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-host-slash\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.162143 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161037    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-host-cni-netd\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.162143 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161071    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.162143 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161099    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xntl6\" (UniqueName: \"kubernetes.io/projected/21f61290-cf4a-4d5b-a1fc-694a52fde254-kube-api-access-xntl6\") pod \"iptables-alerter-pjtfk\" (UID: \"21f61290-cf4a-4d5b-a1fc-694a52fde254\") " pod="openshift-network-operator/iptables-alerter-pjtfk"
Sep 11 17:44:57.162143 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161115    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c8bfa2da-0418-44c2-801b-e5b44882a17b-ovnkube-config\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.162143 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161124    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-etc-kubernetes\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.162143 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161169    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d9qt2\" (UID: \"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8\") " pod="openshift-multus/multus-additional-cni-plugins-d9qt2"
Sep 11 17:44:57.162143 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161184    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-host\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.162143 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161200    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-etc-sysconfig\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.162143 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161209    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-var-lib-openvswitch\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.162143 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161225    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-etc-sysctl-d\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.162143 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161237    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-host-cni-netd\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.162143 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161252    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.162143 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161255    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d9qt2\" (UID: \"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8\") " pod="openshift-multus/multus-additional-cni-plugins-d9qt2"
Sep 11 17:44:57.162143 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161174    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-etc-kubernetes\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.163093 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161355    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/21f61290-cf4a-4d5b-a1fc-694a52fde254-iptables-alerter-script\") pod \"iptables-alerter-pjtfk\" (UID: \"21f61290-cf4a-4d5b-a1fc-694a52fde254\") " pod="openshift-network-operator/iptables-alerter-pjtfk"
Sep 11 17:44:57.163093 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:57.161394    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:44:57.163093 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161576    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a8238ce9-9f28-4900-8b41-667fa71978f2-serviceca\") pod \"node-ca-kpjjv\" (UID: \"a8238ce9-9f28-4900-8b41-667fa71978f2\") " pod="openshift-image-registry/node-ca-kpjjv"
Sep 11 17:44:57.163093 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161579    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-etc-sysconfig\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.163093 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161521    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-run-openvswitch\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.163093 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:57.161662    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8890e4f4-b248-4fe7-b3e7-96fec145ce34-original-pull-secret podName:8890e4f4-b248-4fe7-b3e7-96fec145ce34 nodeName:}" failed. No retries permitted until 2026-09-11 17:44:57.661629317 +0000 UTC m=+3.106729874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8890e4f4-b248-4fe7-b3e7-96fec145ce34-original-pull-secret") pod "global-pull-secret-syncer-rsgml" (UID: "8890e4f4-b248-4fe7-b3e7-96fec145ce34") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:44:57.163093 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161686    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-etc-sysctl-d\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.163093 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161701    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz2ql\" (UniqueName: \"kubernetes.io/projected/d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8-kube-api-access-mz2ql\") pod \"multus-additional-cni-plugins-d9qt2\" (UID: \"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8\") " pod="openshift-multus/multus-additional-cni-plugins-d9qt2"
Sep 11 17:44:57.163093 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161753    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c8bfa2da-0418-44c2-801b-e5b44882a17b-env-overrides\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.163093 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161786    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-etc-sysctl-conf\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.163093 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161813    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-sys\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.163093 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161855    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/500ad4eb-717e-4ab2-879e-ea321c6259f3-tmp\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.163093 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161865    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d9qt2\" (UID: \"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8\") " pod="openshift-multus/multus-additional-cni-plugins-d9qt2"
Sep 11 17:44:57.163093 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161942    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-systemd-units\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.163093 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161970    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-log-socket\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.163093 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.161993    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-host-cni-bin\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.163093 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.162021    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-var-lib-kubelet\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.163985 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.162040    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-etc-sysctl-conf\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.163985 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.162049    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-systemd-units\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.163985 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.162088    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-host-run-netns\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.163985 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.162106    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-var-lib-kubelet\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.163985 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.162118    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-etc-openvswitch\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.163985 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.162150    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/500ad4eb-717e-4ab2-879e-ea321c6259f3-sys\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.163985 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.162162    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-log-socket\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.163985 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.162187    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d9qt2\" (UID: \"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8\") " pod="openshift-multus/multus-additional-cni-plugins-d9qt2"
Sep 11 17:44:57.163985 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.162248    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-host-run-netns\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.163985 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.162252    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c8bfa2da-0418-44c2-801b-e5b44882a17b-env-overrides\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.163985 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.162243    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-etc-openvswitch\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.163985 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.162254    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-host-slash\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.163985 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.162300    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8890e4f4-b248-4fe7-b3e7-96fec145ce34-kubelet-config\") pod \"global-pull-secret-syncer-rsgml\" (UID: \"8890e4f4-b248-4fe7-b3e7-96fec145ce34\") " pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:44:57.163985 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.162389    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8bfa2da-0418-44c2-801b-e5b44882a17b-host-cni-bin\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.163985 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.162638    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-d9qt2\" (UID: \"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8\") " pod="openshift-multus/multus-additional-cni-plugins-d9qt2"
Sep 11 17:44:57.163985 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.162822    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/500ad4eb-717e-4ab2-879e-ea321c6259f3-etc-tuned\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.164787 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.164613    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8bfa2da-0418-44c2-801b-e5b44882a17b-ovn-node-metrics-cert\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.166720 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.166486    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/500ad4eb-717e-4ab2-879e-ea321c6259f3-tmp\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.182796 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.182757    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz2ql\" (UniqueName: \"kubernetes.io/projected/d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8-kube-api-access-mz2ql\") pod \"multus-additional-cni-plugins-d9qt2\" (UID: \"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8\") " pod="openshift-multus/multus-additional-cni-plugins-d9qt2"
Sep 11 17:44:57.184821 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.184797    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bxrk\" (UniqueName: \"kubernetes.io/projected/c8bfa2da-0418-44c2-801b-e5b44882a17b-kube-api-access-8bxrk\") pod \"ovnkube-node-hzjgd\" (UID: \"c8bfa2da-0418-44c2-801b-e5b44882a17b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.186858 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.186823    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v48xf\" (UniqueName: \"kubernetes.io/projected/a8238ce9-9f28-4900-8b41-667fa71978f2-kube-api-access-v48xf\") pod \"node-ca-kpjjv\" (UID: \"a8238ce9-9f28-4900-8b41-667fa71978f2\") " pod="openshift-image-registry/node-ca-kpjjv"
Sep 11 17:44:57.186996 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.186945    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp8bd\" (UniqueName: \"kubernetes.io/projected/500ad4eb-717e-4ab2-879e-ea321c6259f3-kube-api-access-jp8bd\") pod \"tuned-lqlkq\" (UID: \"500ad4eb-717e-4ab2-879e-ea321c6259f3\") " pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.187829 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.187789    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xntl6\" (UniqueName: \"kubernetes.io/projected/21f61290-cf4a-4d5b-a1fc-694a52fde254-kube-api-access-xntl6\") pod \"iptables-alerter-pjtfk\" (UID: \"21f61290-cf4a-4d5b-a1fc-694a52fde254\") " pod="openshift-network-operator/iptables-alerter-pjtfk"
Sep 11 17:44:57.253587 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.253492    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p2znc"
Sep 11 17:44:57.266393 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.266346    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6"
Sep 11 17:44:57.275277 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.275255    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wh2s7"
Sep 11 17:44:57.282602 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.282573    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5qftl"
Sep 11 17:44:57.291469 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.291443    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d9qt2"
Sep 11 17:44:57.300469 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.300441    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lqlkq"
Sep 11 17:44:57.315493 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.315470    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kpjjv"
Sep 11 17:44:57.324303 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.324278    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:44:57.334384 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.334350    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pjtfk"
Sep 11 17:44:57.566523 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.566423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs\") pod \"network-metrics-daemon-wb9dr\" (UID: \"5ee7b9a9-a331-4b43-b68e-439edfc2dfd6\") " pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:44:57.566523 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.566503    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw6c9\" (UniqueName: \"kubernetes.io/projected/f7aa7737-b530-483f-b132-4768683949d3-kube-api-access-lw6c9\") pod \"network-check-target-8nchm\" (UID: \"f7aa7737-b530-483f-b132-4768683949d3\") " pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:44:57.566754 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:57.566603    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:44:57.566754 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:57.566646    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:44:57.566754 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:57.566670    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:44:57.566754 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:57.566683    2559 projected.go:196] Error preparing data for projected volume kube-api-access-lw6c9 for pod openshift-network-diagnostics/network-check-target-8nchm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:44:57.566754 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:57.566693    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs podName:5ee7b9a9-a331-4b43-b68e-439edfc2dfd6 nodeName:}" failed. No retries permitted until 2026-09-11 17:44:58.566669368 +0000 UTC m=+4.011769934 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs") pod "network-metrics-daemon-wb9dr" (UID: "5ee7b9a9-a331-4b43-b68e-439edfc2dfd6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:44:57.566754 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:57.566726    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7aa7737-b530-483f-b132-4768683949d3-kube-api-access-lw6c9 podName:f7aa7737-b530-483f-b132-4768683949d3 nodeName:}" failed. No retries permitted until 2026-09-11 17:44:58.566712411 +0000 UTC m=+4.011812977 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lw6c9" (UniqueName: "kubernetes.io/projected/f7aa7737-b530-483f-b132-4768683949d3-kube-api-access-lw6c9") pod "network-check-target-8nchm" (UID: "f7aa7737-b530-483f-b132-4768683949d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:44:57.666931 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.666881    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8890e4f4-b248-4fe7-b3e7-96fec145ce34-original-pull-secret\") pod \"global-pull-secret-syncer-rsgml\" (UID: \"8890e4f4-b248-4fe7-b3e7-96fec145ce34\") " pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:44:57.667115 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:57.667015    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:44:57.667115 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:57.667106    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8890e4f4-b248-4fe7-b3e7-96fec145ce34-original-pull-secret podName:8890e4f4-b248-4fe7-b3e7-96fec145ce34 nodeName:}" failed. No retries permitted until 2026-09-11 17:44:58.667083934 +0000 UTC m=+4.112184497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8890e4f4-b248-4fe7-b3e7-96fec145ce34-original-pull-secret") pod "global-pull-secret-syncer-rsgml" (UID: "8890e4f4-b248-4fe7-b3e7-96fec145ce34") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:44:57.998153 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.998032    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 17:39:57 +0000 UTC" deadline="2028-03-31 10:05:06.976714242 +0000 UTC"
Sep 11 17:44:57.998153 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:57.998098    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13600h20m8.97862121s"
Sep 11 17:44:58.046712 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:58.046673    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:44:58.046905 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:58.046825    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsgml" podUID="8890e4f4-b248-4fe7-b3e7-96fec145ce34"
Sep 11 17:44:58.405889 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:58.405857    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f2f35a_7ce1_4f08_9ef1_2e9b481141e8.slice/crio-333ba73875a00e8762634c183c4a24b8edda9d5b8774d99feabcf7c51de30e4f WatchSource:0}: Error finding container 333ba73875a00e8762634c183c4a24b8edda9d5b8774d99feabcf7c51de30e4f: Status 404 returned error can't find the container with id 333ba73875a00e8762634c183c4a24b8edda9d5b8774d99feabcf7c51de30e4f
Sep 11 17:44:58.412737 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:58.412702    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f61290_cf4a_4d5b_a1fc_694a52fde254.slice/crio-5d173b353afe729e358d447496a88fa4171cf8b48267db58ece14b1576072dae WatchSource:0}: Error finding container 5d173b353afe729e358d447496a88fa4171cf8b48267db58ece14b1576072dae: Status 404 returned error can't find the container with id 5d173b353afe729e358d447496a88fa4171cf8b48267db58ece14b1576072dae
Sep 11 17:44:58.415812 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:58.415769    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8238ce9_9f28_4900_8b41_667fa71978f2.slice/crio-49bf02d9c2668571eff2d11f72c8c9e3daea098a428f83529e0e35a2381547fb WatchSource:0}: Error finding container 49bf02d9c2668571eff2d11f72c8c9e3daea098a428f83529e0e35a2381547fb: Status 404 returned error can't find the container with id 49bf02d9c2668571eff2d11f72c8c9e3daea098a428f83529e0e35a2381547fb
Sep 11 17:44:58.416743 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:58.416709    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65223d8_9c38_454a_a958_120affb54b81.slice/crio-b13a6e4a3566d2cd568470b2db032087c1222d2362872bcc4ee88664f50b2b39 WatchSource:0}: Error finding container b13a6e4a3566d2cd568470b2db032087c1222d2362872bcc4ee88664f50b2b39: Status 404 returned error can't find the container with id b13a6e4a3566d2cd568470b2db032087c1222d2362872bcc4ee88664f50b2b39
Sep 11 17:44:58.417748 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:58.417628    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500ad4eb_717e_4ab2_879e_ea321c6259f3.slice/crio-b678ccf8ebbd73e17055460160442640bf82ae401f4ae6d44db688e0dd19e713 WatchSource:0}: Error finding container b678ccf8ebbd73e17055460160442640bf82ae401f4ae6d44db688e0dd19e713: Status 404 returned error can't find the container with id b678ccf8ebbd73e17055460160442640bf82ae401f4ae6d44db688e0dd19e713
Sep 11 17:44:58.420306 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:58.419859    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d33fe75_e2cd_43e9_9ab1_0078a32c48cc.slice/crio-7f6f7c83a64e5db195b0030fb7188a1107a29cdaf5ef90666b294e500ec6a62e WatchSource:0}: Error finding container 7f6f7c83a64e5db195b0030fb7188a1107a29cdaf5ef90666b294e500ec6a62e: Status 404 returned error can't find the container with id 7f6f7c83a64e5db195b0030fb7188a1107a29cdaf5ef90666b294e500ec6a62e
Sep 11 17:44:58.420828 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:44:58.420803    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14450ba_245b_4010_9ece_ee27d149274a.slice/crio-597b152e7b332a5159c8c5f7ab1901910d66ffb84075ac1a457c109c356f289a WatchSource:0}: Error finding container 597b152e7b332a5159c8c5f7ab1901910d66ffb84075ac1a457c109c356f289a: Status 404 returned error can't find the container with id 597b152e7b332a5159c8c5f7ab1901910d66ffb84075ac1a457c109c356f289a
Sep 11 17:44:58.574736 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:58.574695    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs\") pod \"network-metrics-daemon-wb9dr\" (UID: \"5ee7b9a9-a331-4b43-b68e-439edfc2dfd6\") " pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:44:58.574904 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:58.574794    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw6c9\" (UniqueName: \"kubernetes.io/projected/f7aa7737-b530-483f-b132-4768683949d3-kube-api-access-lw6c9\") pod \"network-check-target-8nchm\" (UID: \"f7aa7737-b530-483f-b132-4768683949d3\") " pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:44:58.574904 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:58.574852    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:44:58.574994 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:58.574925    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs podName:5ee7b9a9-a331-4b43-b68e-439edfc2dfd6 nodeName:}" failed. No retries permitted until 2026-09-11 17:45:00.574908844 +0000 UTC m=+6.020009412 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs") pod "network-metrics-daemon-wb9dr" (UID: "5ee7b9a9-a331-4b43-b68e-439edfc2dfd6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:44:58.574994 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:58.574929    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:44:58.574994 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:58.574944    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:44:58.574994 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:58.574955    2559 projected.go:196] Error preparing data for projected volume kube-api-access-lw6c9 for pod openshift-network-diagnostics/network-check-target-8nchm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:44:58.575165 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:58.575001    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7aa7737-b530-483f-b132-4768683949d3-kube-api-access-lw6c9 podName:f7aa7737-b530-483f-b132-4768683949d3 nodeName:}" failed. No retries permitted until 2026-09-11 17:45:00.574987803 +0000 UTC m=+6.020088354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lw6c9" (UniqueName: "kubernetes.io/projected/f7aa7737-b530-483f-b132-4768683949d3-kube-api-access-lw6c9") pod "network-check-target-8nchm" (UID: "f7aa7737-b530-483f-b132-4768683949d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:44:58.676240 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:58.676048    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8890e4f4-b248-4fe7-b3e7-96fec145ce34-original-pull-secret\") pod \"global-pull-secret-syncer-rsgml\" (UID: \"8890e4f4-b248-4fe7-b3e7-96fec145ce34\") " pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:44:58.676320 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:58.676193    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:44:58.676372 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:58.676342    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8890e4f4-b248-4fe7-b3e7-96fec145ce34-original-pull-secret podName:8890e4f4-b248-4fe7-b3e7-96fec145ce34 nodeName:}" failed. No retries permitted until 2026-09-11 17:45:00.676307784 +0000 UTC m=+6.121408338 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8890e4f4-b248-4fe7-b3e7-96fec145ce34-original-pull-secret") pod "global-pull-secret-syncer-rsgml" (UID: "8890e4f4-b248-4fe7-b3e7-96fec145ce34") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:44:58.998803 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:58.998726    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 17:39:57 +0000 UTC" deadline="2028-06-09 03:45:24.958854616 +0000 UTC"
Sep 11 17:44:58.998803 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:58.998801    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15274h0m25.960058338s"
Sep 11 17:44:59.050741 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:59.049939    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:44:59.050741 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:59.050072    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nchm" podUID="f7aa7737-b530-483f-b132-4768683949d3"
Sep 11 17:44:59.050741 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:59.050562    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:44:59.050741 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:44:59.050680    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb9dr" podUID="5ee7b9a9-a331-4b43-b68e-439edfc2dfd6"
Sep 11 17:44:59.064971 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:59.063583    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kpjjv" event={"ID":"a8238ce9-9f28-4900-8b41-667fa71978f2","Type":"ContainerStarted","Data":"49bf02d9c2668571eff2d11f72c8c9e3daea098a428f83529e0e35a2381547fb"}
Sep 11 17:44:59.068841 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:59.068752    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pjtfk" event={"ID":"21f61290-cf4a-4d5b-a1fc-694a52fde254","Type":"ContainerStarted","Data":"5d173b353afe729e358d447496a88fa4171cf8b48267db58ece14b1576072dae"}
Sep 11 17:44:59.073883 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:59.072859    2559 generic.go:356] "Generic (PLEG): container finished" podID="6194dcd4cf42d1c4c056092b6c620840" containerID="a0c894dde1264fa8a60329370343f211536a35df39dcb001cc40e7e7247cfcbb" exitCode=0
Sep 11 17:44:59.073883 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:59.072947    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-96.ec2.internal" event={"ID":"6194dcd4cf42d1c4c056092b6c620840","Type":"ContainerDied","Data":"a0c894dde1264fa8a60329370343f211536a35df39dcb001cc40e7e7247cfcbb"}
Sep 11 17:44:59.078956 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:59.078463    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-96.ec2.internal" event={"ID":"aa3822d42868560dd3e2a5d526ebe096","Type":"ContainerStarted","Data":"a4265a790927edaf0cd680b056544279abfb48c1b368bb08872da964f2efeb21"}
Sep 11 17:44:59.085729 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:59.085687    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wh2s7" event={"ID":"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc","Type":"ContainerStarted","Data":"7f6f7c83a64e5db195b0030fb7188a1107a29cdaf5ef90666b294e500ec6a62e"}
Sep 11 17:44:59.096287 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:59.096252    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lqlkq" event={"ID":"500ad4eb-717e-4ab2-879e-ea321c6259f3","Type":"ContainerStarted","Data":"b678ccf8ebbd73e17055460160442640bf82ae401f4ae6d44db688e0dd19e713"}
Sep 11 17:44:59.107894 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:59.104926    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd" event={"ID":"c8bfa2da-0418-44c2-801b-e5b44882a17b","Type":"ContainerStarted","Data":"5b063eb70eb9c4d2e3ddc34e3d79b7b9f9d7177aafcbbc7798286172ca96ba94"}
Sep 11 17:44:59.115086 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:59.114996    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5qftl" event={"ID":"190c54c5-ae20-4ef1-b9b2-af966c1712f4","Type":"ContainerStarted","Data":"1d12ded0a76a425ec58d993fb0b60641273d2d83091525ef7792ed6b8fc350d6"}
Sep 11 17:44:59.123029 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:59.122991    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d9qt2" event={"ID":"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8","Type":"ContainerStarted","Data":"333ba73875a00e8762634c183c4a24b8edda9d5b8774d99feabcf7c51de30e4f"}
Sep 11 17:44:59.131038 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:59.131005    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6" event={"ID":"c14450ba-245b-4010-9ece-ee27d149274a","Type":"ContainerStarted","Data":"597b152e7b332a5159c8c5f7ab1901910d66ffb84075ac1a457c109c356f289a"}
Sep 11 17:44:59.133004 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:44:59.132972    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p2znc" event={"ID":"a65223d8-9c38-454a-a958-120affb54b81","Type":"ContainerStarted","Data":"b13a6e4a3566d2cd568470b2db032087c1222d2362872bcc4ee88664f50b2b39"}
Sep 11 17:45:00.046909 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:00.046810    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:45:00.047411 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:00.046963    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsgml" podUID="8890e4f4-b248-4fe7-b3e7-96fec145ce34"
Sep 11 17:45:00.157506 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:00.157467    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-96.ec2.internal" event={"ID":"6194dcd4cf42d1c4c056092b6c620840","Type":"ContainerStarted","Data":"a8362aa642e71659b4114be0c047c3cb983b546da4cfb6671e7a47d2c468f3d6"}
Sep 11 17:45:00.173988 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:00.173915    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-96.ec2.internal" podStartSLOduration=4.17389029 podStartE2EDuration="4.17389029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:44:59.10891056 +0000 UTC m=+4.554011138" watchObservedRunningTime="2026-09-11 17:45:00.17389029 +0000 UTC m=+5.618990865"
Sep 11 17:45:00.593580 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:00.593544    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs\") pod \"network-metrics-daemon-wb9dr\" (UID: \"5ee7b9a9-a331-4b43-b68e-439edfc2dfd6\") " pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:45:00.593767 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:00.593629    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw6c9\" (UniqueName: \"kubernetes.io/projected/f7aa7737-b530-483f-b132-4768683949d3-kube-api-access-lw6c9\") pod \"network-check-target-8nchm\" (UID: \"f7aa7737-b530-483f-b132-4768683949d3\") " pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:45:00.593846 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:00.593801    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:45:00.593846 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:00.593831    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:45:00.593846 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:00.593844    2559 projected.go:196] Error preparing data for projected volume kube-api-access-lw6c9 for pod openshift-network-diagnostics/network-check-target-8nchm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:45:00.594011 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:00.593906    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7aa7737-b530-483f-b132-4768683949d3-kube-api-access-lw6c9 podName:f7aa7737-b530-483f-b132-4768683949d3 nodeName:}" failed. No retries permitted until 2026-09-11 17:45:04.593886681 +0000 UTC m=+10.038987249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lw6c9" (UniqueName: "kubernetes.io/projected/f7aa7737-b530-483f-b132-4768683949d3-kube-api-access-lw6c9") pod "network-check-target-8nchm" (UID: "f7aa7737-b530-483f-b132-4768683949d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:45:00.594644 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:00.594618    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:45:00.594760 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:00.594683    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs podName:5ee7b9a9-a331-4b43-b68e-439edfc2dfd6 nodeName:}" failed. No retries permitted until 2026-09-11 17:45:04.594665397 +0000 UTC m=+10.039765949 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs") pod "network-metrics-daemon-wb9dr" (UID: "5ee7b9a9-a331-4b43-b68e-439edfc2dfd6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:45:00.694976 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:00.694937    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8890e4f4-b248-4fe7-b3e7-96fec145ce34-original-pull-secret\") pod \"global-pull-secret-syncer-rsgml\" (UID: \"8890e4f4-b248-4fe7-b3e7-96fec145ce34\") " pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:45:00.695177 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:00.695122    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:45:00.695250 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:00.695189    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8890e4f4-b248-4fe7-b3e7-96fec145ce34-original-pull-secret podName:8890e4f4-b248-4fe7-b3e7-96fec145ce34 nodeName:}" failed. No retries permitted until 2026-09-11 17:45:04.695170597 +0000 UTC m=+10.140271151 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8890e4f4-b248-4fe7-b3e7-96fec145ce34-original-pull-secret") pod "global-pull-secret-syncer-rsgml" (UID: "8890e4f4-b248-4fe7-b3e7-96fec145ce34") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:45:01.048583 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:01.048550    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:45:01.049063 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:01.048694    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nchm" podUID="f7aa7737-b530-483f-b132-4768683949d3"
Sep 11 17:45:01.050775 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:01.050748    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:45:01.050916 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:01.050874    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb9dr" podUID="5ee7b9a9-a331-4b43-b68e-439edfc2dfd6"
Sep 11 17:45:02.047219 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:02.047180    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:45:02.047484 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:02.047326    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsgml" podUID="8890e4f4-b248-4fe7-b3e7-96fec145ce34"
Sep 11 17:45:03.047052 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:03.047014    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:45:03.047711 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:03.047073    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:45:03.047711 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:03.047174    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nchm" podUID="f7aa7737-b530-483f-b132-4768683949d3"
Sep 11 17:45:03.047711 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:03.047306    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb9dr" podUID="5ee7b9a9-a331-4b43-b68e-439edfc2dfd6"
Sep 11 17:45:04.046831 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:04.046750    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:45:04.046998 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:04.046907    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsgml" podUID="8890e4f4-b248-4fe7-b3e7-96fec145ce34"
Sep 11 17:45:04.631041 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:04.630617    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw6c9\" (UniqueName: \"kubernetes.io/projected/f7aa7737-b530-483f-b132-4768683949d3-kube-api-access-lw6c9\") pod \"network-check-target-8nchm\" (UID: \"f7aa7737-b530-483f-b132-4768683949d3\") " pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:45:04.631041 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:04.630707    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs\") pod \"network-metrics-daemon-wb9dr\" (UID: \"5ee7b9a9-a331-4b43-b68e-439edfc2dfd6\") " pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:45:04.631041 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:04.630782    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:45:04.631041 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:04.630792    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:45:04.631041 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:04.630803    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:45:04.631041 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:04.630815    2559 projected.go:196] Error preparing data for projected volume kube-api-access-lw6c9 for pod openshift-network-diagnostics/network-check-target-8nchm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:45:04.631041 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:04.630860    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs podName:5ee7b9a9-a331-4b43-b68e-439edfc2dfd6 nodeName:}" failed. No retries permitted until 2026-09-11 17:45:12.630839408 +0000 UTC m=+18.075939974 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs") pod "network-metrics-daemon-wb9dr" (UID: "5ee7b9a9-a331-4b43-b68e-439edfc2dfd6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:45:04.631041 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:04.630878    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7aa7737-b530-483f-b132-4768683949d3-kube-api-access-lw6c9 podName:f7aa7737-b530-483f-b132-4768683949d3 nodeName:}" failed. No retries permitted until 2026-09-11 17:45:12.630868908 +0000 UTC m=+18.075969456 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lw6c9" (UniqueName: "kubernetes.io/projected/f7aa7737-b530-483f-b132-4768683949d3-kube-api-access-lw6c9") pod "network-check-target-8nchm" (UID: "f7aa7737-b530-483f-b132-4768683949d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:45:04.731901 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:04.731339    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8890e4f4-b248-4fe7-b3e7-96fec145ce34-original-pull-secret\") pod \"global-pull-secret-syncer-rsgml\" (UID: \"8890e4f4-b248-4fe7-b3e7-96fec145ce34\") " pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:45:04.731901 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:04.731484    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:45:04.731901 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:04.731563    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8890e4f4-b248-4fe7-b3e7-96fec145ce34-original-pull-secret podName:8890e4f4-b248-4fe7-b3e7-96fec145ce34 nodeName:}" failed. No retries permitted until 2026-09-11 17:45:12.731543568 +0000 UTC m=+18.176644142 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8890e4f4-b248-4fe7-b3e7-96fec145ce34-original-pull-secret") pod "global-pull-secret-syncer-rsgml" (UID: "8890e4f4-b248-4fe7-b3e7-96fec145ce34") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:45:05.047481 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:05.047446    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:45:05.047658 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:05.047571    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nchm" podUID="f7aa7737-b530-483f-b132-4768683949d3"
Sep 11 17:45:05.047752 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:05.047722    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:45:05.047876 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:05.047853    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb9dr" podUID="5ee7b9a9-a331-4b43-b68e-439edfc2dfd6"
Sep 11 17:45:06.046871 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:06.046844    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:45:06.047344 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:06.046963    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsgml" podUID="8890e4f4-b248-4fe7-b3e7-96fec145ce34"
Sep 11 17:45:07.047506 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:07.047457    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:45:07.047990 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:07.047597    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb9dr" podUID="5ee7b9a9-a331-4b43-b68e-439edfc2dfd6"
Sep 11 17:45:07.047990 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:07.047456    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:45:07.048110 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:07.048062    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nchm" podUID="f7aa7737-b530-483f-b132-4768683949d3"
Sep 11 17:45:08.046611 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:08.046579    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:45:08.046774 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:08.046683    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsgml" podUID="8890e4f4-b248-4fe7-b3e7-96fec145ce34"
Sep 11 17:45:09.047459 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:09.047429    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:45:09.047777 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:09.047557    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nchm" podUID="f7aa7737-b530-483f-b132-4768683949d3"
Sep 11 17:45:09.047777 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:09.047605    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:45:09.047777 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:09.047721    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb9dr" podUID="5ee7b9a9-a331-4b43-b68e-439edfc2dfd6"
Sep 11 17:45:10.047362 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:10.047315    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:45:10.047540 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:10.047464    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsgml" podUID="8890e4f4-b248-4fe7-b3e7-96fec145ce34"
Sep 11 17:45:11.046731 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:11.046646    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:45:11.046914 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:11.046841    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:45:11.050271 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:11.047090    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb9dr" podUID="5ee7b9a9-a331-4b43-b68e-439edfc2dfd6"
Sep 11 17:45:11.050271 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:11.047446    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nchm" podUID="f7aa7737-b530-483f-b132-4768683949d3"
Sep 11 17:45:12.047091 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:12.047054    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:45:12.047312 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:12.047194    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsgml" podUID="8890e4f4-b248-4fe7-b3e7-96fec145ce34"
Sep 11 17:45:12.692539 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:12.692499    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw6c9\" (UniqueName: \"kubernetes.io/projected/f7aa7737-b530-483f-b132-4768683949d3-kube-api-access-lw6c9\") pod \"network-check-target-8nchm\" (UID: \"f7aa7737-b530-483f-b132-4768683949d3\") " pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:45:12.693066 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:12.692582    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs\") pod \"network-metrics-daemon-wb9dr\" (UID: \"5ee7b9a9-a331-4b43-b68e-439edfc2dfd6\") " pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:45:12.693066 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:12.692700    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:45:12.693066 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:12.692707    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:45:12.693066 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:12.692740    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:45:12.693066 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:12.692750    2559 projected.go:196] Error preparing data for projected volume kube-api-access-lw6c9 for pod openshift-network-diagnostics/network-check-target-8nchm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:45:12.693066 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:12.692774    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs podName:5ee7b9a9-a331-4b43-b68e-439edfc2dfd6 nodeName:}" failed. No retries permitted until 2026-09-11 17:45:28.69275443 +0000 UTC m=+34.137854993 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs") pod "network-metrics-daemon-wb9dr" (UID: "5ee7b9a9-a331-4b43-b68e-439edfc2dfd6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:45:12.693066 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:12.692793    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7aa7737-b530-483f-b132-4768683949d3-kube-api-access-lw6c9 podName:f7aa7737-b530-483f-b132-4768683949d3 nodeName:}" failed. No retries permitted until 2026-09-11 17:45:28.692782058 +0000 UTC m=+34.137882623 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lw6c9" (UniqueName: "kubernetes.io/projected/f7aa7737-b530-483f-b132-4768683949d3-kube-api-access-lw6c9") pod "network-check-target-8nchm" (UID: "f7aa7737-b530-483f-b132-4768683949d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:45:12.793202 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:12.793156    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8890e4f4-b248-4fe7-b3e7-96fec145ce34-original-pull-secret\") pod \"global-pull-secret-syncer-rsgml\" (UID: \"8890e4f4-b248-4fe7-b3e7-96fec145ce34\") " pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:45:12.793423 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:12.793293    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:45:12.793423 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:12.793384    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8890e4f4-b248-4fe7-b3e7-96fec145ce34-original-pull-secret podName:8890e4f4-b248-4fe7-b3e7-96fec145ce34 nodeName:}" failed. No retries permitted until 2026-09-11 17:45:28.793364095 +0000 UTC m=+34.238464658 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8890e4f4-b248-4fe7-b3e7-96fec145ce34-original-pull-secret") pod "global-pull-secret-syncer-rsgml" (UID: "8890e4f4-b248-4fe7-b3e7-96fec145ce34") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:45:13.047370 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:13.047258    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:45:13.047545 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:13.047258    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:45:13.047545 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:13.047436    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb9dr" podUID="5ee7b9a9-a331-4b43-b68e-439edfc2dfd6"
Sep 11 17:45:13.047545 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:13.047456    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nchm" podUID="f7aa7737-b530-483f-b132-4768683949d3"
Sep 11 17:45:14.046838 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:14.046804    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:45:14.047234 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:14.046925    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsgml" podUID="8890e4f4-b248-4fe7-b3e7-96fec145ce34"
Sep 11 17:45:15.047395 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:15.047345    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:45:15.047395 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:15.047388    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:45:15.047874 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:15.047459    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb9dr" podUID="5ee7b9a9-a331-4b43-b68e-439edfc2dfd6"
Sep 11 17:45:15.047874 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:15.047591    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nchm" podUID="f7aa7737-b530-483f-b132-4768683949d3"
Sep 11 17:45:16.047231 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:16.047181    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:45:16.047484 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:16.047309    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsgml" podUID="8890e4f4-b248-4fe7-b3e7-96fec145ce34"
Sep 11 17:45:17.047427 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:17.047092    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:45:17.047596 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:17.047133    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:45:17.047596 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:17.047538    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb9dr" podUID="5ee7b9a9-a331-4b43-b68e-439edfc2dfd6"
Sep 11 17:45:17.047596 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:17.047569    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nchm" podUID="f7aa7737-b530-483f-b132-4768683949d3"
Sep 11 17:45:17.199760 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:17.199732    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wh2s7" event={"ID":"7d33fe75-e2cd-43e9-9ab1-0078a32c48cc","Type":"ContainerStarted","Data":"db8ef5ad6c66aedc0b13ecf8c98d9b431611fdcf4fc2bd7dd8535f06ca0f5941"}
Sep 11 17:45:17.203039 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:17.203011    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lqlkq" event={"ID":"500ad4eb-717e-4ab2-879e-ea321c6259f3","Type":"ContainerStarted","Data":"bbeca62a6867031f1a3f8363cd9efb9e734c82e26ba9b2242842689ab59b3fcc"}
Sep 11 17:45:17.205743 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:17.205720    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd" event={"ID":"c8bfa2da-0418-44c2-801b-e5b44882a17b","Type":"ContainerStarted","Data":"da04a17f7582a26569e9d56ecf7401f5bfc4671ad941950c4b8736c95c6466d6"}
Sep 11 17:45:17.205873 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:17.205780    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd" event={"ID":"c8bfa2da-0418-44c2-801b-e5b44882a17b","Type":"ContainerStarted","Data":"919526d446f101e4752b0c6b01eb749573f1c47e40ad79262c097b8c2c03da10"}
Sep 11 17:45:17.205873 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:17.205822    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd" event={"ID":"c8bfa2da-0418-44c2-801b-e5b44882a17b","Type":"ContainerStarted","Data":"62227ab81425d5b7de0c2ed34fabd6a09d93e567efba15ad2b81cc9795b77246"}
Sep 11 17:45:17.205873 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:17.205853    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd" event={"ID":"c8bfa2da-0418-44c2-801b-e5b44882a17b","Type":"ContainerStarted","Data":"1e19a8a922f365e44be59bf96a2f1feb940f592636b2fa5317a8f32e63d58619"}
Sep 11 17:45:17.206000 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:17.205887    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd" event={"ID":"c8bfa2da-0418-44c2-801b-e5b44882a17b","Type":"ContainerStarted","Data":"f5a3f6b31d9e950b2b9e4b8b43580ce5126b20feb901ba8658efaf9bb51618bf"}
Sep 11 17:45:17.207081 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:17.207060    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5qftl" event={"ID":"190c54c5-ae20-4ef1-b9b2-af966c1712f4","Type":"ContainerStarted","Data":"607a3ce9115492b416ed608f96126847f5002ca9db4cf66d4a8ee086ad8b766d"}
Sep 11 17:45:17.208570 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:17.208547    2559 generic.go:356] "Generic (PLEG): container finished" podID="d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8" containerID="1f6af5b534f0a42b35e09a2d04b99e41e46efc37cd9b850456c399ad8f806d9f" exitCode=0
Sep 11 17:45:17.208658 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:17.208612    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d9qt2" event={"ID":"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8","Type":"ContainerDied","Data":"1f6af5b534f0a42b35e09a2d04b99e41e46efc37cd9b850456c399ad8f806d9f"}
Sep 11 17:45:17.210165 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:17.210133    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6" event={"ID":"c14450ba-245b-4010-9ece-ee27d149274a","Type":"ContainerStarted","Data":"4ac08cc6a2a5843fed5ae72d3b22b6479710025e377ece6d77a861a4fa3da384"}
Sep 11 17:45:17.211571 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:17.211549    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p2znc" event={"ID":"a65223d8-9c38-454a-a958-120affb54b81","Type":"ContainerStarted","Data":"5db56ace498e1860d6928fd413178490ac89fac662d3bb05de68b3e34d0093f1"}
Sep 11 17:45:17.212898 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:17.212878    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kpjjv" event={"ID":"a8238ce9-9f28-4900-8b41-667fa71978f2","Type":"ContainerStarted","Data":"9cf767444a4ab4d738ed11a83f5f5acdbbb7d66ceae0b2c2760befd5f35483b8"}
Sep 11 17:45:17.218548 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:17.218503    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-96.ec2.internal" podStartSLOduration=21.218492381 podStartE2EDuration="21.218492381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:45:00.174630563 +0000 UTC m=+5.619731139" watchObservedRunningTime="2026-09-11 17:45:17.218492381 +0000 UTC m=+22.663592953"
Sep 11 17:45:17.218796 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:17.218776    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wh2s7" podStartSLOduration=4.076044575 podStartE2EDuration="22.218769241s" totalImagesPullingTime="18.142724666s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:45:17.218619443 +0000 UTC m=+22.663720005" watchObservedRunningTime="2026-09-11 17:45:17.218769241 +0000 UTC m=+22.663869845"
Sep 11 17:45:17.234333 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:17.234293    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5qftl" podStartSLOduration=4.150851987 podStartE2EDuration="22.234278964s" totalImagesPullingTime="18.083426977s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:45:17.234058816 +0000 UTC m=+22.679159390" watchObservedRunningTime="2026-09-11 17:45:17.234278964 +0000 UTC m=+22.679379534"
Sep 11 17:45:17.277495 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:17.277450    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lqlkq" podStartSLOduration=3.202084975 podStartE2EDuration="21.277433428s" totalImagesPullingTime="18.075348453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:44:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:45:17.277237703 +0000 UTC m=+22.722338275" watchObservedRunningTime="2026-09-11 17:45:17.277433428 +0000 UTC m=+22.722533998"
Sep 11 17:45:17.294737 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:17.294691    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-p2znc" podStartSLOduration=8.353098017 podStartE2EDuration="22.294674118s" totalImagesPullingTime="13.941576101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:45:17.294230275 +0000 UTC m=+22.739330846" watchObservedRunningTime="2026-09-11 17:45:17.294674118 +0000 UTC m=+22.739774703"
Sep 11 17:45:17.611706 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:17.611665    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-p2znc"
Sep 11 17:45:17.612166 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:17.612149    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-p2znc"
Sep 11 17:45:17.633390 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:17.632048    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kpjjv" podStartSLOduration=3.55652332 podStartE2EDuration="21.632026848s" totalImagesPullingTime="18.075503528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:44:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:45:17.318546131 +0000 UTC m=+22.763646702" watchObservedRunningTime="2026-09-11 17:45:17.632026848 +0000 UTC m=+23.077127421"
Sep 11 17:45:18.047227 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:18.047191    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:45:18.047387 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:18.047296    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsgml" podUID="8890e4f4-b248-4fe7-b3e7-96fec145ce34"
Sep 11 17:45:18.219011 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:18.218928    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd" event={"ID":"c8bfa2da-0418-44c2-801b-e5b44882a17b","Type":"ContainerStarted","Data":"30cd1990f5bcf513dfce8495ba89bd125f61cd4aeef32169a66f84f58ad168e4"}
Sep 11 17:45:18.221165 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:18.220758    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pjtfk" event={"ID":"21f61290-cf4a-4d5b-a1fc-694a52fde254","Type":"ContainerStarted","Data":"c5b2f67242592e14a96a4a665a7821dd62fcdf401884c956236ed7eb47ac6a3c"}
Sep 11 17:45:18.221517 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:18.221192    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-p2znc"
Sep 11 17:45:18.221650 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:18.221626    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-p2znc"
Sep 11 17:45:18.238701 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:18.238657    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pjtfk" podStartSLOduration=4.160023127 podStartE2EDuration="22.238627286s" totalImagesPullingTime="18.078604159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:44:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:45:18.237936531 +0000 UTC m=+23.683037112" watchObservedRunningTime="2026-09-11 17:45:18.238627286 +0000 UTC m=+23.683727858"
Sep 11 17:45:18.374784 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:18.374759    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 17:45:19.013240 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:19.013123    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T17:45:18.374777325Z","UUID":"458183b7-cee5-4d66-acd0-21ca98a893ae","Handler":null,"Name":"","Endpoint":""}
Sep 11 17:45:19.015247 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:19.015220    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 17:45:19.015412 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:19.015261    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 17:45:19.046502 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:19.046474    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:45:19.046673 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:19.046594    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nchm" podUID="f7aa7737-b530-483f-b132-4768683949d3"
Sep 11 17:45:19.046974 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:19.046952    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:45:19.047073 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:19.047057    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb9dr" podUID="5ee7b9a9-a331-4b43-b68e-439edfc2dfd6"
Sep 11 17:45:19.226166 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:19.226125    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6" event={"ID":"c14450ba-245b-4010-9ece-ee27d149274a","Type":"ContainerStarted","Data":"14184627ff08d0df4de083328b3457a0a2d27ab26898518b395fa62511022bfc"}
Sep 11 17:45:20.047707 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:20.047466    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:45:20.047854 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:20.047760    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsgml" podUID="8890e4f4-b248-4fe7-b3e7-96fec145ce34"
Sep 11 17:45:20.232215 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:20.232181    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd" event={"ID":"c8bfa2da-0418-44c2-801b-e5b44882a17b","Type":"ContainerStarted","Data":"6d8fbc8a4c21192f42203ba3ca28b6b24ae000093b805f65680dd11fa7747f1e"}
Sep 11 17:45:20.234424 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:20.234390    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6" event={"ID":"c14450ba-245b-4010-9ece-ee27d149274a","Type":"ContainerStarted","Data":"c4ad8051837b465aded88c73b5ae399c69206de0f0562222bbc25239423b5b79"}
Sep 11 17:45:20.252729 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:20.252676    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fkqb6" podStartSLOduration=4.411851543 podStartE2EDuration="25.252659069s" totalImagesPullingTime="20.840807526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:44:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:45:20.252117786 +0000 UTC m=+25.697218356" watchObservedRunningTime="2026-09-11 17:45:20.252659069 +0000 UTC m=+25.697759640"
Sep 11 17:45:21.047257 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:21.047215    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:45:21.047463 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:21.047374    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb9dr" podUID="5ee7b9a9-a331-4b43-b68e-439edfc2dfd6"
Sep 11 17:45:21.047463 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:21.047424    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:45:21.047599 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:21.047545    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nchm" podUID="f7aa7737-b530-483f-b132-4768683949d3"
Sep 11 17:45:22.047527 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:22.047272    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:45:22.048111 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:22.047547    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsgml" podUID="8890e4f4-b248-4fe7-b3e7-96fec145ce34"
Sep 11 17:45:22.242110 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:22.242071    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd" event={"ID":"c8bfa2da-0418-44c2-801b-e5b44882a17b","Type":"ContainerStarted","Data":"87996bcbd12eb2b63a0202adfa10815b0459e941f5d7134de60c1297612f19b5"}
Sep 11 17:45:22.242390 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:22.242372    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:45:22.244018 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:22.243986    2559 generic.go:356] "Generic (PLEG): container finished" podID="d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8" containerID="ce8331834e209386f3c3af232d21f263dd5c9c99d431fa76aa95a2f4229a2ab8" exitCode=0
Sep 11 17:45:22.244111 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:22.244053    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d9qt2" event={"ID":"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8","Type":"ContainerDied","Data":"ce8331834e209386f3c3af232d21f263dd5c9c99d431fa76aa95a2f4229a2ab8"}
Sep 11 17:45:22.257933 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:22.257911    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:45:22.322505 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:22.322409    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd" podStartSLOduration=8.18458391 podStartE2EDuration="26.322393397s" totalImagesPullingTime="18.137809487s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:44:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:45:22.322141189 +0000 UTC m=+27.767241759" watchObservedRunningTime="2026-09-11 17:45:22.322393397 +0000 UTC m=+27.767493967"
Sep 11 17:45:23.049785 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:23.049589    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:45:23.049785 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:23.049721    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb9dr" podUID="5ee7b9a9-a331-4b43-b68e-439edfc2dfd6"
Sep 11 17:45:23.050299 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:23.050121    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:45:23.050299 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:23.050211    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nchm" podUID="f7aa7737-b530-483f-b132-4768683949d3"
Sep 11 17:45:23.249049 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:23.249015    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d9qt2" event={"ID":"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8","Type":"ContainerStarted","Data":"34d28a09666932e783b1066c8a11d859c4d6dc7a85127cb3e31b6736e70b3589"}
Sep 11 17:45:23.249682 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:23.249661    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:45:23.249808 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:23.249724    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:45:23.264713 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:23.264683    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:45:23.840415 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:23.838876    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rsgml"]
Sep 11 17:45:23.840415 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:23.839090    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:45:23.840415 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:23.839246    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsgml" podUID="8890e4f4-b248-4fe7-b3e7-96fec145ce34"
Sep 11 17:45:23.842578 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:23.842518    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8nchm"]
Sep 11 17:45:23.842685 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:23.842605    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:45:23.842685 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:23.842673    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nchm" podUID="f7aa7737-b530-483f-b132-4768683949d3"
Sep 11 17:45:23.855663 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:23.855638    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wb9dr"]
Sep 11 17:45:23.855793 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:23.855758    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:45:23.855860 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:23.855841    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb9dr" podUID="5ee7b9a9-a331-4b43-b68e-439edfc2dfd6"
Sep 11 17:45:24.254421 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:24.254218    2559 generic.go:356] "Generic (PLEG): container finished" podID="d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8" containerID="34d28a09666932e783b1066c8a11d859c4d6dc7a85127cb3e31b6736e70b3589" exitCode=0
Sep 11 17:45:24.254421 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:24.254418    2559 generic.go:356] "Generic (PLEG): container finished" podID="d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8" containerID="68193893c9b4260a301078bc5de13b3eb205810814212b2a92904d4de480aafe" exitCode=0
Sep 11 17:45:24.254863 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:24.254296    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d9qt2" event={"ID":"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8","Type":"ContainerDied","Data":"34d28a09666932e783b1066c8a11d859c4d6dc7a85127cb3e31b6736e70b3589"}
Sep 11 17:45:24.254863 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:24.254539    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d9qt2" event={"ID":"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8","Type":"ContainerDied","Data":"68193893c9b4260a301078bc5de13b3eb205810814212b2a92904d4de480aafe"}
Sep 11 17:45:26.046996 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:26.046957    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:45:26.046996 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:26.046987    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:45:26.047512 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:26.046970    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:45:26.047512 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:26.047080    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nchm" podUID="f7aa7737-b530-483f-b132-4768683949d3"
Sep 11 17:45:26.047512 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:26.047171    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsgml" podUID="8890e4f4-b248-4fe7-b3e7-96fec145ce34"
Sep 11 17:45:26.047512 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:26.047254    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb9dr" podUID="5ee7b9a9-a331-4b43-b68e-439edfc2dfd6"
Sep 11 17:45:28.046724 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:28.046688    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:45:28.046724 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:28.046710    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:45:28.047380 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:28.046688    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:45:28.047380 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:28.046817    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb9dr" podUID="5ee7b9a9-a331-4b43-b68e-439edfc2dfd6"
Sep 11 17:45:28.047380 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:28.046890    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nchm" podUID="f7aa7737-b530-483f-b132-4768683949d3"
Sep 11 17:45:28.047380 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:28.046987    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsgml" podUID="8890e4f4-b248-4fe7-b3e7-96fec145ce34"
Sep 11 17:45:28.705925 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:28.705885    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs\") pod \"network-metrics-daemon-wb9dr\" (UID: \"5ee7b9a9-a331-4b43-b68e-439edfc2dfd6\") " pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:45:28.706126 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:28.705956    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw6c9\" (UniqueName: \"kubernetes.io/projected/f7aa7737-b530-483f-b132-4768683949d3-kube-api-access-lw6c9\") pod \"network-check-target-8nchm\" (UID: \"f7aa7737-b530-483f-b132-4768683949d3\") " pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:45:28.706126 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:28.706070    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:45:28.706233 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:28.706132    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:45:28.706233 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:28.706152    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs podName:5ee7b9a9-a331-4b43-b68e-439edfc2dfd6 nodeName:}" failed. No retries permitted until 2026-09-11 17:46:00.706127557 +0000 UTC m=+66.151228127 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs") pod "network-metrics-daemon-wb9dr" (UID: "5ee7b9a9-a331-4b43-b68e-439edfc2dfd6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:45:28.706233 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:28.706158    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:45:28.706233 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:28.706174    2559 projected.go:196] Error preparing data for projected volume kube-api-access-lw6c9 for pod openshift-network-diagnostics/network-check-target-8nchm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:45:28.706233 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:28.706229    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7aa7737-b530-483f-b132-4768683949d3-kube-api-access-lw6c9 podName:f7aa7737-b530-483f-b132-4768683949d3 nodeName:}" failed. No retries permitted until 2026-09-11 17:46:00.706214631 +0000 UTC m=+66.151315180 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lw6c9" (UniqueName: "kubernetes.io/projected/f7aa7737-b530-483f-b132-4768683949d3-kube-api-access-lw6c9") pod "network-check-target-8nchm" (UID: "f7aa7737-b530-483f-b132-4768683949d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:45:28.807243 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:28.807210    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8890e4f4-b248-4fe7-b3e7-96fec145ce34-original-pull-secret\") pod \"global-pull-secret-syncer-rsgml\" (UID: \"8890e4f4-b248-4fe7-b3e7-96fec145ce34\") " pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:45:28.807399 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:28.807339    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:45:28.807471 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:28.807411    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8890e4f4-b248-4fe7-b3e7-96fec145ce34-original-pull-secret podName:8890e4f4-b248-4fe7-b3e7-96fec145ce34 nodeName:}" failed. No retries permitted until 2026-09-11 17:46:00.807398931 +0000 UTC m=+66.252499493 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8890e4f4-b248-4fe7-b3e7-96fec145ce34-original-pull-secret") pod "global-pull-secret-syncer-rsgml" (UID: "8890e4f4-b248-4fe7-b3e7-96fec145ce34") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:45:30.047237 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.047202    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:45:30.047237 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.047230    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:45:30.047778 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.047202    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:45:30.047778 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:30.047326    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb9dr" podUID="5ee7b9a9-a331-4b43-b68e-439edfc2dfd6"
Sep 11 17:45:30.047778 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:30.047452    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nchm" podUID="f7aa7737-b530-483f-b132-4768683949d3"
Sep 11 17:45:30.047778 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:30.047562    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rsgml" podUID="8890e4f4-b248-4fe7-b3e7-96fec145ce34"
Sep 11 17:45:30.271739 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.271657    2559 generic.go:356] "Generic (PLEG): container finished" podID="d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8" containerID="e3d9d828d9b2ca25fbe6705ce42d8e110322df2ee39dc8c5c2b971fe5657d8e4" exitCode=0
Sep 11 17:45:30.271739 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.271706    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d9qt2" event={"ID":"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8","Type":"ContainerDied","Data":"e3d9d828d9b2ca25fbe6705ce42d8e110322df2ee39dc8c5c2b971fe5657d8e4"}
Sep 11 17:45:30.357191 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.357165    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-96.ec2.internal" event="NodeReady"
Sep 11 17:45:30.357360 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.357311    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 17:45:30.454998 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.454968    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-h8grx"]
Sep 11 17:45:30.463773 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.463750    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cspnd"]
Sep 11 17:45:30.463895 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.463876    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h8grx"
Sep 11 17:45:30.469217 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.469196    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 17:45:30.469320 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.469228    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tc7ww\""
Sep 11 17:45:30.469320 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.469205    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 17:45:30.478096 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.478072    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cspnd"
Sep 11 17:45:30.481077 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.481057    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 17:45:30.481199 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.481181    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5xw8q\""
Sep 11 17:45:30.481658 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.481643    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 17:45:30.481963 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.481951    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 17:45:30.485117 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.485071    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cspnd"]
Sep 11 17:45:30.488425 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.488401    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h8grx"]
Sep 11 17:45:30.520146 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.520124    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdcrz\" (UniqueName: \"kubernetes.io/projected/e831333a-4c9b-4a40-a53e-c66f3e2f653f-kube-api-access-wdcrz\") pod \"dns-default-h8grx\" (UID: \"e831333a-4c9b-4a40-a53e-c66f3e2f653f\") " pod="openshift-dns/dns-default-h8grx"
Sep 11 17:45:30.520276 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.520151    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e831333a-4c9b-4a40-a53e-c66f3e2f653f-metrics-tls\") pod \"dns-default-h8grx\" (UID: \"e831333a-4c9b-4a40-a53e-c66f3e2f653f\") " pod="openshift-dns/dns-default-h8grx"
Sep 11 17:45:30.520276 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.520170    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e831333a-4c9b-4a40-a53e-c66f3e2f653f-config-volume\") pod \"dns-default-h8grx\" (UID: \"e831333a-4c9b-4a40-a53e-c66f3e2f653f\") " pod="openshift-dns/dns-default-h8grx"
Sep 11 17:45:30.520409 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.520284    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e831333a-4c9b-4a40-a53e-c66f3e2f653f-tmp-dir\") pod \"dns-default-h8grx\" (UID: \"e831333a-4c9b-4a40-a53e-c66f3e2f653f\") " pod="openshift-dns/dns-default-h8grx"
Sep 11 17:45:30.621537 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.621450    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e831333a-4c9b-4a40-a53e-c66f3e2f653f-config-volume\") pod \"dns-default-h8grx\" (UID: \"e831333a-4c9b-4a40-a53e-c66f3e2f653f\") " pod="openshift-dns/dns-default-h8grx"
Sep 11 17:45:30.621537 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.621513    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx99f\" (UniqueName: \"kubernetes.io/projected/37764305-549b-498a-967e-c24cbc84e019-kube-api-access-gx99f\") pod \"ingress-canary-cspnd\" (UID: \"37764305-549b-498a-967e-c24cbc84e019\") " pod="openshift-ingress-canary/ingress-canary-cspnd"
Sep 11 17:45:30.621722 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.621540    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e831333a-4c9b-4a40-a53e-c66f3e2f653f-tmp-dir\") pod \"dns-default-h8grx\" (UID: \"e831333a-4c9b-4a40-a53e-c66f3e2f653f\") " pod="openshift-dns/dns-default-h8grx"
Sep 11 17:45:30.621722 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.621565    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37764305-549b-498a-967e-c24cbc84e019-cert\") pod \"ingress-canary-cspnd\" (UID: \"37764305-549b-498a-967e-c24cbc84e019\") " pod="openshift-ingress-canary/ingress-canary-cspnd"
Sep 11 17:45:30.621722 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.621591    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdcrz\" (UniqueName: \"kubernetes.io/projected/e831333a-4c9b-4a40-a53e-c66f3e2f653f-kube-api-access-wdcrz\") pod \"dns-default-h8grx\" (UID: \"e831333a-4c9b-4a40-a53e-c66f3e2f653f\") " pod="openshift-dns/dns-default-h8grx"
Sep 11 17:45:30.621722 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.621609    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e831333a-4c9b-4a40-a53e-c66f3e2f653f-metrics-tls\") pod \"dns-default-h8grx\" (UID: \"e831333a-4c9b-4a40-a53e-c66f3e2f653f\") " pod="openshift-dns/dns-default-h8grx"
Sep 11 17:45:30.621722 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:30.621708    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:45:30.621885 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:30.621762    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e831333a-4c9b-4a40-a53e-c66f3e2f653f-metrics-tls podName:e831333a-4c9b-4a40-a53e-c66f3e2f653f nodeName:}" failed. No retries permitted until 2026-09-11 17:45:31.121748157 +0000 UTC m=+36.566848712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e831333a-4c9b-4a40-a53e-c66f3e2f653f-metrics-tls") pod "dns-default-h8grx" (UID: "e831333a-4c9b-4a40-a53e-c66f3e2f653f") : secret "dns-default-metrics-tls" not found
Sep 11 17:45:30.621885 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.621836    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e831333a-4c9b-4a40-a53e-c66f3e2f653f-tmp-dir\") pod \"dns-default-h8grx\" (UID: \"e831333a-4c9b-4a40-a53e-c66f3e2f653f\") " pod="openshift-dns/dns-default-h8grx"
Sep 11 17:45:30.622114 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.622094    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e831333a-4c9b-4a40-a53e-c66f3e2f653f-config-volume\") pod \"dns-default-h8grx\" (UID: \"e831333a-4c9b-4a40-a53e-c66f3e2f653f\") " pod="openshift-dns/dns-default-h8grx"
Sep 11 17:45:30.639121 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.639085    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdcrz\" (UniqueName: \"kubernetes.io/projected/e831333a-4c9b-4a40-a53e-c66f3e2f653f-kube-api-access-wdcrz\") pod \"dns-default-h8grx\" (UID: \"e831333a-4c9b-4a40-a53e-c66f3e2f653f\") " pod="openshift-dns/dns-default-h8grx"
Sep 11 17:45:30.722096 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.722057    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37764305-549b-498a-967e-c24cbc84e019-cert\") pod \"ingress-canary-cspnd\" (UID: \"37764305-549b-498a-967e-c24cbc84e019\") " pod="openshift-ingress-canary/ingress-canary-cspnd"
Sep 11 17:45:30.722261 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.722139    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx99f\" (UniqueName: \"kubernetes.io/projected/37764305-549b-498a-967e-c24cbc84e019-kube-api-access-gx99f\") pod \"ingress-canary-cspnd\" (UID: \"37764305-549b-498a-967e-c24cbc84e019\") " pod="openshift-ingress-canary/ingress-canary-cspnd"
Sep 11 17:45:30.724523 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.724498    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37764305-549b-498a-967e-c24cbc84e019-cert\") pod \"ingress-canary-cspnd\" (UID: \"37764305-549b-498a-967e-c24cbc84e019\") " pod="openshift-ingress-canary/ingress-canary-cspnd"
Sep 11 17:45:30.731857 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.731836    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx99f\" (UniqueName: \"kubernetes.io/projected/37764305-549b-498a-967e-c24cbc84e019-kube-api-access-gx99f\") pod \"ingress-canary-cspnd\" (UID: \"37764305-549b-498a-967e-c24cbc84e019\") " pod="openshift-ingress-canary/ingress-canary-cspnd"
Sep 11 17:45:30.823009 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:30.822968    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cspnd"
Sep 11 17:45:31.125589 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:31.125412    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e831333a-4c9b-4a40-a53e-c66f3e2f653f-metrics-tls\") pod \"dns-default-h8grx\" (UID: \"e831333a-4c9b-4a40-a53e-c66f3e2f653f\") " pod="openshift-dns/dns-default-h8grx"
Sep 11 17:45:31.126142 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:31.125560    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:45:31.126142 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:31.125663    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e831333a-4c9b-4a40-a53e-c66f3e2f653f-metrics-tls podName:e831333a-4c9b-4a40-a53e-c66f3e2f653f nodeName:}" failed. No retries permitted until 2026-09-11 17:45:32.125640447 +0000 UTC m=+37.570740999 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e831333a-4c9b-4a40-a53e-c66f3e2f653f-metrics-tls") pod "dns-default-h8grx" (UID: "e831333a-4c9b-4a40-a53e-c66f3e2f653f") : secret "dns-default-metrics-tls" not found
Sep 11 17:45:31.149255 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:31.149218    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cspnd"]
Sep 11 17:45:31.153766 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:45:31.153737    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37764305_549b_498a_967e_c24cbc84e019.slice/crio-55bcc853bd0c26352de2d64d226c82bb0339031f72917cd3cadc49a85867852d WatchSource:0}: Error finding container 55bcc853bd0c26352de2d64d226c82bb0339031f72917cd3cadc49a85867852d: Status 404 returned error can't find the container with id 55bcc853bd0c26352de2d64d226c82bb0339031f72917cd3cadc49a85867852d
Sep 11 17:45:31.280197 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:31.280069    2559 generic.go:356] "Generic (PLEG): container finished" podID="d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8" containerID="a003a572c791b1677689149e9b98847ab126d119af7893d3d1ab20184752c381" exitCode=0
Sep 11 17:45:31.280197 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:31.280139    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d9qt2" event={"ID":"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8","Type":"ContainerDied","Data":"a003a572c791b1677689149e9b98847ab126d119af7893d3d1ab20184752c381"}
Sep 11 17:45:31.281428 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:31.281402    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cspnd" event={"ID":"37764305-549b-498a-967e-c24cbc84e019","Type":"ContainerStarted","Data":"55bcc853bd0c26352de2d64d226c82bb0339031f72917cd3cadc49a85867852d"}
Sep 11 17:45:32.046706 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:32.046672    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:45:32.046886 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:32.046675    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:45:32.046886 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:32.046684    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:45:32.055252 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:32.054686    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 17:45:32.055651 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:32.055625    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jfwbq\""
Sep 11 17:45:32.055783 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:32.055630    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 17:45:32.055783 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:32.055685    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 17:45:32.055783 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:32.055625    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7dtkl\""
Sep 11 17:45:32.055783 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:32.055741    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 17:45:32.134953 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:32.134913    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e831333a-4c9b-4a40-a53e-c66f3e2f653f-metrics-tls\") pod \"dns-default-h8grx\" (UID: \"e831333a-4c9b-4a40-a53e-c66f3e2f653f\") " pod="openshift-dns/dns-default-h8grx"
Sep 11 17:45:32.135388 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:32.135066    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:45:32.135388 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:32.135140    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e831333a-4c9b-4a40-a53e-c66f3e2f653f-metrics-tls podName:e831333a-4c9b-4a40-a53e-c66f3e2f653f nodeName:}" failed. No retries permitted until 2026-09-11 17:45:34.135119549 +0000 UTC m=+39.580220112 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e831333a-4c9b-4a40-a53e-c66f3e2f653f-metrics-tls") pod "dns-default-h8grx" (UID: "e831333a-4c9b-4a40-a53e-c66f3e2f653f") : secret "dns-default-metrics-tls" not found
Sep 11 17:45:32.288866 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:32.288828    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d9qt2" event={"ID":"d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8","Type":"ContainerStarted","Data":"ed85f25834e51a67e0316b746cfd3fa61f59917747cb9f8dddf540c56e43035c"}
Sep 11 17:45:32.319513 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:32.318855    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d9qt2" podStartSLOduration=7.838451659 podStartE2EDuration="37.318833681s" totalImagesPullingTime="29.480382022s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:44:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:45:32.318394397 +0000 UTC m=+37.763494965" watchObservedRunningTime="2026-09-11 17:45:32.318833681 +0000 UTC m=+37.763934255"
Sep 11 17:45:34.151194 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:34.151011    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e831333a-4c9b-4a40-a53e-c66f3e2f653f-metrics-tls\") pod \"dns-default-h8grx\" (UID: \"e831333a-4c9b-4a40-a53e-c66f3e2f653f\") " pod="openshift-dns/dns-default-h8grx"
Sep 11 17:45:34.151608 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:34.151164    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:45:34.151608 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:34.151259    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e831333a-4c9b-4a40-a53e-c66f3e2f653f-metrics-tls podName:e831333a-4c9b-4a40-a53e-c66f3e2f653f nodeName:}" failed. No retries permitted until 2026-09-11 17:45:38.151245674 +0000 UTC m=+43.596346236 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e831333a-4c9b-4a40-a53e-c66f3e2f653f-metrics-tls") pod "dns-default-h8grx" (UID: "e831333a-4c9b-4a40-a53e-c66f3e2f653f") : secret "dns-default-metrics-tls" not found
Sep 11 17:45:34.295638 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:34.295596    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cspnd" event={"ID":"37764305-549b-498a-967e-c24cbc84e019","Type":"ContainerStarted","Data":"21094782cbd87b44337bd40949358e62e78254b214a98a00668d4f6015e91509"}
Sep 11 17:45:34.326922 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:34.326866    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cspnd" podStartSLOduration=1.9196828510000001 podStartE2EDuration="4.326851101s" totalImagesPullingTime="2.40716825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:45:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:45:34.32633129 +0000 UTC m=+39.771431861" watchObservedRunningTime="2026-09-11 17:45:34.326851101 +0000 UTC m=+39.771951710"
Sep 11 17:45:38.177593 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:38.177545    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e831333a-4c9b-4a40-a53e-c66f3e2f653f-metrics-tls\") pod \"dns-default-h8grx\" (UID: \"e831333a-4c9b-4a40-a53e-c66f3e2f653f\") " pod="openshift-dns/dns-default-h8grx"
Sep 11 17:45:38.178081 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:38.177681    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:45:38.178081 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:38.177738    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e831333a-4c9b-4a40-a53e-c66f3e2f653f-metrics-tls podName:e831333a-4c9b-4a40-a53e-c66f3e2f653f nodeName:}" failed. No retries permitted until 2026-09-11 17:45:46.177723313 +0000 UTC m=+51.622823881 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e831333a-4c9b-4a40-a53e-c66f3e2f653f-metrics-tls") pod "dns-default-h8grx" (UID: "e831333a-4c9b-4a40-a53e-c66f3e2f653f") : secret "dns-default-metrics-tls" not found
Sep 11 17:45:46.227879 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:46.227844    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e831333a-4c9b-4a40-a53e-c66f3e2f653f-metrics-tls\") pod \"dns-default-h8grx\" (UID: \"e831333a-4c9b-4a40-a53e-c66f3e2f653f\") " pod="openshift-dns/dns-default-h8grx"
Sep 11 17:45:46.228294 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:46.227962    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:45:46.228294 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:45:46.228017    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e831333a-4c9b-4a40-a53e-c66f3e2f653f-metrics-tls podName:e831333a-4c9b-4a40-a53e-c66f3e2f653f nodeName:}" failed. No retries permitted until 2026-09-11 17:46:02.22800083 +0000 UTC m=+67.673101399 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e831333a-4c9b-4a40-a53e-c66f3e2f653f-metrics-tls") pod "dns-default-h8grx" (UID: "e831333a-4c9b-4a40-a53e-c66f3e2f653f") : secret "dns-default-metrics-tls" not found
Sep 11 17:45:55.267700 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:45:55.267673    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hzjgd"
Sep 11 17:46:00.730001 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:00.729953    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw6c9\" (UniqueName: \"kubernetes.io/projected/f7aa7737-b530-483f-b132-4768683949d3-kube-api-access-lw6c9\") pod \"network-check-target-8nchm\" (UID: \"f7aa7737-b530-483f-b132-4768683949d3\") " pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:46:00.730528 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:00.730015    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs\") pod \"network-metrics-daemon-wb9dr\" (UID: \"5ee7b9a9-a331-4b43-b68e-439edfc2dfd6\") " pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:46:00.732539 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:00.732520    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 17:46:00.732642 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:00.732623    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 17:46:00.741037 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:00.741014    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 17:46:00.741093 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:00.741083    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs podName:5ee7b9a9-a331-4b43-b68e-439edfc2dfd6 nodeName:}" failed. No retries permitted until 2026-09-11 17:47:04.741065369 +0000 UTC m=+130.186165923 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs") pod "network-metrics-daemon-wb9dr" (UID: "5ee7b9a9-a331-4b43-b68e-439edfc2dfd6") : secret "metrics-daemon-secret" not found
Sep 11 17:46:00.743700 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:00.743680    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 17:46:00.755687 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:00.755659    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw6c9\" (UniqueName: \"kubernetes.io/projected/f7aa7737-b530-483f-b132-4768683949d3-kube-api-access-lw6c9\") pod \"network-check-target-8nchm\" (UID: \"f7aa7737-b530-483f-b132-4768683949d3\") " pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:46:00.830847 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:00.830795    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8890e4f4-b248-4fe7-b3e7-96fec145ce34-original-pull-secret\") pod \"global-pull-secret-syncer-rsgml\" (UID: \"8890e4f4-b248-4fe7-b3e7-96fec145ce34\") " pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:46:00.833755 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:00.833737    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 17:46:00.843977 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:00.843954    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8890e4f4-b248-4fe7-b3e7-96fec145ce34-original-pull-secret\") pod \"global-pull-secret-syncer-rsgml\" (UID: \"8890e4f4-b248-4fe7-b3e7-96fec145ce34\") " pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:46:00.861549 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:00.861523    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7dtkl\""
Sep 11 17:46:00.869201 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:00.869185    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:46:00.871358 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:00.871332    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rsgml"
Sep 11 17:46:01.148363 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:01.148308    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rsgml"]
Sep 11 17:46:01.149180 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:01.149135    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8nchm"]
Sep 11 17:46:01.151420 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:46:01.151387    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7aa7737_b530_483f_b132_4768683949d3.slice/crio-d45ea452246a71a126b6354414328fe77cc0bda8d412c61c579ebe49b878b10e WatchSource:0}: Error finding container d45ea452246a71a126b6354414328fe77cc0bda8d412c61c579ebe49b878b10e: Status 404 returned error can't find the container with id d45ea452246a71a126b6354414328fe77cc0bda8d412c61c579ebe49b878b10e
Sep 11 17:46:01.151852 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:46:01.151832    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8890e4f4_b248_4fe7_b3e7_96fec145ce34.slice/crio-e7e83ddbfff896f51f99e472b5caa7a8a8d7c2bb1782f5f389f78844cb7d0268 WatchSource:0}: Error finding container e7e83ddbfff896f51f99e472b5caa7a8a8d7c2bb1782f5f389f78844cb7d0268: Status 404 returned error can't find the container with id e7e83ddbfff896f51f99e472b5caa7a8a8d7c2bb1782f5f389f78844cb7d0268
Sep 11 17:46:01.359290 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:01.359252    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rsgml" event={"ID":"8890e4f4-b248-4fe7-b3e7-96fec145ce34","Type":"ContainerStarted","Data":"e7e83ddbfff896f51f99e472b5caa7a8a8d7c2bb1782f5f389f78844cb7d0268"}
Sep 11 17:46:01.360404 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:01.360377    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8nchm" event={"ID":"f7aa7737-b530-483f-b132-4768683949d3","Type":"ContainerStarted","Data":"d45ea452246a71a126b6354414328fe77cc0bda8d412c61c579ebe49b878b10e"}
Sep 11 17:46:02.241669 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:02.241627    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e831333a-4c9b-4a40-a53e-c66f3e2f653f-metrics-tls\") pod \"dns-default-h8grx\" (UID: \"e831333a-4c9b-4a40-a53e-c66f3e2f653f\") " pod="openshift-dns/dns-default-h8grx"
Sep 11 17:46:02.242147 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:02.241812    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:46:02.242147 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:02.241899    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e831333a-4c9b-4a40-a53e-c66f3e2f653f-metrics-tls podName:e831333a-4c9b-4a40-a53e-c66f3e2f653f nodeName:}" failed. No retries permitted until 2026-09-11 17:46:34.241877873 +0000 UTC m=+99.686978436 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e831333a-4c9b-4a40-a53e-c66f3e2f653f-metrics-tls") pod "dns-default-h8grx" (UID: "e831333a-4c9b-4a40-a53e-c66f3e2f653f") : secret "dns-default-metrics-tls" not found
Sep 11 17:46:06.376000 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:06.375959    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rsgml" event={"ID":"8890e4f4-b248-4fe7-b3e7-96fec145ce34","Type":"ContainerStarted","Data":"889e8e894a624a554bef9f54877f0d5ffc0eb4406a2f45ce0827425e5c9f2e3b"}
Sep 11 17:46:06.377416 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:06.377392    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8nchm" event={"ID":"f7aa7737-b530-483f-b132-4768683949d3","Type":"ContainerStarted","Data":"36812c80b5fd1fd77ca7af86e82c37078c0ff8d1f02931b312603eade095a09e"}
Sep 11 17:46:06.377552 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:06.377447    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:46:06.412998 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:06.412948    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rsgml" podStartSLOduration=65.816460984 podStartE2EDuration="1m10.412929381s" totalImagesPullingTime="4.596468397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:44:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:46:06.395335466 +0000 UTC m=+71.840436047" watchObservedRunningTime="2026-09-11 17:46:06.412929381 +0000 UTC m=+71.858029932"
Sep 11 17:46:06.413769 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:06.413729    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8nchm" podStartSLOduration=66.820334073 podStartE2EDuration="1m11.413717768s" totalImagesPullingTime="4.593383695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:46:06.413008818 +0000 UTC m=+71.858109401" watchObservedRunningTime="2026-09-11 17:46:06.413717768 +0000 UTC m=+71.858818338"
Sep 11 17:46:14.982176 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:14.982142    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-lcltf"]
Sep 11 17:46:14.986278 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:14.986260    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-lcltf"
Sep 11 17:46:14.989806 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:14.989777    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-68f69558bb-s4pcz"]
Sep 11 17:46:14.991818 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:14.991794    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 11 17:46:14.991943 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:14.991831    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 17:46:14.991943 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:14.991901    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 17:46:14.992757 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:14.992661    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:14.992973 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:14.992952    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hn98l\""
Sep 11 17:46:14.993220 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:14.993200    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 11 17:46:14.997696 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:14.997675    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 11 17:46:14.997876 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:14.997853    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 11 17:46:14.997985 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:14.997942    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 11 17:46:14.998158 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:14.998140    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 17:46:14.998291 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:14.998272    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 17:46:14.998442 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:14.998165    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 11 17:46:14.999229 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:14.999210    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lzs5h\""
Sep 11 17:46:14.999318 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:14.999271    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-lcltf"]
Sep 11 17:46:15.004977 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.004952    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 11 17:46:15.032733 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.032699    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/15f610bc-3cc9-46de-b61c-bae98f80781f-default-certificate\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:15.032733 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.032736    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15f610bc-3cc9-46de-b61c-bae98f80781f-metrics-certs\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:15.032940 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.032754    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c539f9b8-d329-47d0-a8ca-79ad211a80c3-serving-cert\") pod \"insights-operator-5844b856db-lcltf\" (UID: \"c539f9b8-d329-47d0-a8ca-79ad211a80c3\") " pod="openshift-insights/insights-operator-5844b856db-lcltf"
Sep 11 17:46:15.032940 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.032772    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c539f9b8-d329-47d0-a8ca-79ad211a80c3-service-ca-bundle\") pod \"insights-operator-5844b856db-lcltf\" (UID: \"c539f9b8-d329-47d0-a8ca-79ad211a80c3\") " pod="openshift-insights/insights-operator-5844b856db-lcltf"
Sep 11 17:46:15.032940 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.032814    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kwbq\" (UniqueName: \"kubernetes.io/projected/c539f9b8-d329-47d0-a8ca-79ad211a80c3-kube-api-access-9kwbq\") pod \"insights-operator-5844b856db-lcltf\" (UID: \"c539f9b8-d329-47d0-a8ca-79ad211a80c3\") " pod="openshift-insights/insights-operator-5844b856db-lcltf"
Sep 11 17:46:15.032940 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.032855    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm4j8\" (UniqueName: \"kubernetes.io/projected/15f610bc-3cc9-46de-b61c-bae98f80781f-kube-api-access-pm4j8\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:15.032940 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.032878    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c539f9b8-d329-47d0-a8ca-79ad211a80c3-tmp\") pod \"insights-operator-5844b856db-lcltf\" (UID: \"c539f9b8-d329-47d0-a8ca-79ad211a80c3\") " pod="openshift-insights/insights-operator-5844b856db-lcltf"
Sep 11 17:46:15.032940 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.032895    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c539f9b8-d329-47d0-a8ca-79ad211a80c3-snapshots\") pod \"insights-operator-5844b856db-lcltf\" (UID: \"c539f9b8-d329-47d0-a8ca-79ad211a80c3\") " pod="openshift-insights/insights-operator-5844b856db-lcltf"
Sep 11 17:46:15.032940 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.032909    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c539f9b8-d329-47d0-a8ca-79ad211a80c3-trusted-ca-bundle\") pod \"insights-operator-5844b856db-lcltf\" (UID: \"c539f9b8-d329-47d0-a8ca-79ad211a80c3\") " pod="openshift-insights/insights-operator-5844b856db-lcltf"
Sep 11 17:46:15.032940 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.032930    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f610bc-3cc9-46de-b61c-bae98f80781f-service-ca-bundle\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:15.033173 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.032993    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/15f610bc-3cc9-46de-b61c-bae98f80781f-stats-auth\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:15.121715 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.121680    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-68f69558bb-s4pcz"]
Sep 11 17:46:15.133344 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.133308    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/15f610bc-3cc9-46de-b61c-bae98f80781f-stats-auth\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:15.133522 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.133400    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/15f610bc-3cc9-46de-b61c-bae98f80781f-default-certificate\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:15.133522 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.133437    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15f610bc-3cc9-46de-b61c-bae98f80781f-metrics-certs\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:15.133522 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.133459    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c539f9b8-d329-47d0-a8ca-79ad211a80c3-serving-cert\") pod \"insights-operator-5844b856db-lcltf\" (UID: \"c539f9b8-d329-47d0-a8ca-79ad211a80c3\") " pod="openshift-insights/insights-operator-5844b856db-lcltf"
Sep 11 17:46:15.133692 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:15.133575    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 17:46:15.133692 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.133596    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c539f9b8-d329-47d0-a8ca-79ad211a80c3-service-ca-bundle\") pod \"insights-operator-5844b856db-lcltf\" (UID: \"c539f9b8-d329-47d0-a8ca-79ad211a80c3\") " pod="openshift-insights/insights-operator-5844b856db-lcltf"
Sep 11 17:46:15.133692 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.133638    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kwbq\" (UniqueName: \"kubernetes.io/projected/c539f9b8-d329-47d0-a8ca-79ad211a80c3-kube-api-access-9kwbq\") pod \"insights-operator-5844b856db-lcltf\" (UID: \"c539f9b8-d329-47d0-a8ca-79ad211a80c3\") " pod="openshift-insights/insights-operator-5844b856db-lcltf"
Sep 11 17:46:15.133692 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:15.133667    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15f610bc-3cc9-46de-b61c-bae98f80781f-metrics-certs podName:15f610bc-3cc9-46de-b61c-bae98f80781f nodeName:}" failed. No retries permitted until 2026-09-11 17:46:15.633643963 +0000 UTC m=+81.078744531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15f610bc-3cc9-46de-b61c-bae98f80781f-metrics-certs") pod "router-default-68f69558bb-s4pcz" (UID: "15f610bc-3cc9-46de-b61c-bae98f80781f") : secret "router-metrics-certs-default" not found
Sep 11 17:46:15.133884 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.133718    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm4j8\" (UniqueName: \"kubernetes.io/projected/15f610bc-3cc9-46de-b61c-bae98f80781f-kube-api-access-pm4j8\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:15.134060 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.134041    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c539f9b8-d329-47d0-a8ca-79ad211a80c3-tmp\") pod \"insights-operator-5844b856db-lcltf\" (UID: \"c539f9b8-d329-47d0-a8ca-79ad211a80c3\") " pod="openshift-insights/insights-operator-5844b856db-lcltf"
Sep 11 17:46:15.134178 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.134075    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c539f9b8-d329-47d0-a8ca-79ad211a80c3-snapshots\") pod \"insights-operator-5844b856db-lcltf\" (UID: \"c539f9b8-d329-47d0-a8ca-79ad211a80c3\") " pod="openshift-insights/insights-operator-5844b856db-lcltf"
Sep 11 17:46:15.134178 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.134100    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c539f9b8-d329-47d0-a8ca-79ad211a80c3-trusted-ca-bundle\") pod \"insights-operator-5844b856db-lcltf\" (UID: \"c539f9b8-d329-47d0-a8ca-79ad211a80c3\") " pod="openshift-insights/insights-operator-5844b856db-lcltf"
Sep 11 17:46:15.134178 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.134148    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f610bc-3cc9-46de-b61c-bae98f80781f-service-ca-bundle\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:15.134379 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:15.134324    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15f610bc-3cc9-46de-b61c-bae98f80781f-service-ca-bundle podName:15f610bc-3cc9-46de-b61c-bae98f80781f nodeName:}" failed. No retries permitted until 2026-09-11 17:46:15.63430525 +0000 UTC m=+81.079405816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/15f610bc-3cc9-46de-b61c-bae98f80781f-service-ca-bundle") pod "router-default-68f69558bb-s4pcz" (UID: "15f610bc-3cc9-46de-b61c-bae98f80781f") : configmap references non-existent config key: service-ca.crt
Sep 11 17:46:15.134452 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.134384    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c539f9b8-d329-47d0-a8ca-79ad211a80c3-service-ca-bundle\") pod \"insights-operator-5844b856db-lcltf\" (UID: \"c539f9b8-d329-47d0-a8ca-79ad211a80c3\") " pod="openshift-insights/insights-operator-5844b856db-lcltf"
Sep 11 17:46:15.134814 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.134786    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c539f9b8-d329-47d0-a8ca-79ad211a80c3-tmp\") pod \"insights-operator-5844b856db-lcltf\" (UID: \"c539f9b8-d329-47d0-a8ca-79ad211a80c3\") " pod="openshift-insights/insights-operator-5844b856db-lcltf"
Sep 11 17:46:15.134968 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.134952    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c539f9b8-d329-47d0-a8ca-79ad211a80c3-trusted-ca-bundle\") pod \"insights-operator-5844b856db-lcltf\" (UID: \"c539f9b8-d329-47d0-a8ca-79ad211a80c3\") " pod="openshift-insights/insights-operator-5844b856db-lcltf"
Sep 11 17:46:15.135241 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.135217    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c539f9b8-d329-47d0-a8ca-79ad211a80c3-snapshots\") pod \"insights-operator-5844b856db-lcltf\" (UID: \"c539f9b8-d329-47d0-a8ca-79ad211a80c3\") " pod="openshift-insights/insights-operator-5844b856db-lcltf"
Sep 11 17:46:15.136226 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.136204    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c539f9b8-d329-47d0-a8ca-79ad211a80c3-serving-cert\") pod \"insights-operator-5844b856db-lcltf\" (UID: \"c539f9b8-d329-47d0-a8ca-79ad211a80c3\") " pod="openshift-insights/insights-operator-5844b856db-lcltf"
Sep 11 17:46:15.136376 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.136260    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/15f610bc-3cc9-46de-b61c-bae98f80781f-default-certificate\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:15.136444 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.136424    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/15f610bc-3cc9-46de-b61c-bae98f80781f-stats-auth\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:15.147284 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.147257    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm4j8\" (UniqueName: \"kubernetes.io/projected/15f610bc-3cc9-46de-b61c-bae98f80781f-kube-api-access-pm4j8\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:15.147707 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.147689    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kwbq\" (UniqueName: \"kubernetes.io/projected/c539f9b8-d329-47d0-a8ca-79ad211a80c3-kube-api-access-9kwbq\") pod \"insights-operator-5844b856db-lcltf\" (UID: \"c539f9b8-d329-47d0-a8ca-79ad211a80c3\") " pod="openshift-insights/insights-operator-5844b856db-lcltf"
Sep 11 17:46:15.193504 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.193468    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4hw4h"]
Sep 11 17:46:15.197598 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.197574    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-5vb7x"]
Sep 11 17:46:15.197712 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.197693    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4hw4h"
Sep 11 17:46:15.200773 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.200751    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 11 17:46:15.200979 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.200963    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 11 17:46:15.201071 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.200975    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-phvdh\""
Sep 11 17:46:15.201250 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.201229    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5vb7x"
Sep 11 17:46:15.201296 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.201257    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:46:15.201457 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.201230    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 11 17:46:15.205133 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.205115    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 11 17:46:15.205221 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.205146    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 11 17:46:15.206087 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.206068    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:46:15.206650 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.206630    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-778l5\""
Sep 11 17:46:15.206733 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.206633    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 11 17:46:15.215299 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.215275    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-5vb7x"]
Sep 11 17:46:15.217972 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.217935    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4hw4h"]
Sep 11 17:46:15.235395 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.235288    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jvsb\" (UniqueName: \"kubernetes.io/projected/abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2-kube-api-access-8jvsb\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4hw4h\" (UID: \"abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4hw4h"
Sep 11 17:46:15.235395 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.235368    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/285adb5b-133b-49bd-a8ff-aaf67c7a3cc6-config\") pod \"service-ca-operator-cf7c776b7-5vb7x\" (UID: \"285adb5b-133b-49bd-a8ff-aaf67c7a3cc6\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5vb7x"
Sep 11 17:46:15.235572 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.235441    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/285adb5b-133b-49bd-a8ff-aaf67c7a3cc6-serving-cert\") pod \"service-ca-operator-cf7c776b7-5vb7x\" (UID: \"285adb5b-133b-49bd-a8ff-aaf67c7a3cc6\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5vb7x"
Sep 11 17:46:15.235572 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.235469    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2kv8\" (UniqueName: \"kubernetes.io/projected/285adb5b-133b-49bd-a8ff-aaf67c7a3cc6-kube-api-access-p2kv8\") pod \"service-ca-operator-cf7c776b7-5vb7x\" (UID: \"285adb5b-133b-49bd-a8ff-aaf67c7a3cc6\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5vb7x"
Sep 11 17:46:15.235572 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.235487    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4hw4h\" (UID: \"abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4hw4h"
Sep 11 17:46:15.235572 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.235505    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4hw4h\" (UID: \"abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4hw4h"
Sep 11 17:46:15.295873 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.295829    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-lcltf"
Sep 11 17:46:15.336769 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.336731    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jvsb\" (UniqueName: \"kubernetes.io/projected/abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2-kube-api-access-8jvsb\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4hw4h\" (UID: \"abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4hw4h"
Sep 11 17:46:15.336934 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.336892    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/285adb5b-133b-49bd-a8ff-aaf67c7a3cc6-config\") pod \"service-ca-operator-cf7c776b7-5vb7x\" (UID: \"285adb5b-133b-49bd-a8ff-aaf67c7a3cc6\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5vb7x"
Sep 11 17:46:15.336980 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.336967    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/285adb5b-133b-49bd-a8ff-aaf67c7a3cc6-serving-cert\") pod \"service-ca-operator-cf7c776b7-5vb7x\" (UID: \"285adb5b-133b-49bd-a8ff-aaf67c7a3cc6\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5vb7x"
Sep 11 17:46:15.337011 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.336991    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2kv8\" (UniqueName: \"kubernetes.io/projected/285adb5b-133b-49bd-a8ff-aaf67c7a3cc6-kube-api-access-p2kv8\") pod \"service-ca-operator-cf7c776b7-5vb7x\" (UID: \"285adb5b-133b-49bd-a8ff-aaf67c7a3cc6\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5vb7x"
Sep 11 17:46:15.337011 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.337007    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4hw4h\" (UID: \"abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4hw4h"
Sep 11 17:46:15.337098 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.337032    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4hw4h\" (UID: \"abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4hw4h"
Sep 11 17:46:15.337630 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.337594    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/285adb5b-133b-49bd-a8ff-aaf67c7a3cc6-config\") pod \"service-ca-operator-cf7c776b7-5vb7x\" (UID: \"285adb5b-133b-49bd-a8ff-aaf67c7a3cc6\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5vb7x"
Sep 11 17:46:15.337753 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.337639    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4hw4h\" (UID: \"abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4hw4h"
Sep 11 17:46:15.339242 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.339197    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/285adb5b-133b-49bd-a8ff-aaf67c7a3cc6-serving-cert\") pod \"service-ca-operator-cf7c776b7-5vb7x\" (UID: \"285adb5b-133b-49bd-a8ff-aaf67c7a3cc6\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5vb7x"
Sep 11 17:46:15.339242 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.339215    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4hw4h\" (UID: \"abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4hw4h"
Sep 11 17:46:15.345703 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.345678    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2kv8\" (UniqueName: \"kubernetes.io/projected/285adb5b-133b-49bd-a8ff-aaf67c7a3cc6-kube-api-access-p2kv8\") pod \"service-ca-operator-cf7c776b7-5vb7x\" (UID: \"285adb5b-133b-49bd-a8ff-aaf67c7a3cc6\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5vb7x"
Sep 11 17:46:15.346422 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.346400    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jvsb\" (UniqueName: \"kubernetes.io/projected/abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2-kube-api-access-8jvsb\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4hw4h\" (UID: \"abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4hw4h"
Sep 11 17:46:15.508485 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.508378    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4hw4h"
Sep 11 17:46:15.513323 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.513305    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5vb7x"
Sep 11 17:46:15.569765 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.569531    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-lcltf"]
Sep 11 17:46:15.639648 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.639608    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15f610bc-3cc9-46de-b61c-bae98f80781f-metrics-certs\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:15.639850 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.639660    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f610bc-3cc9-46de-b61c-bae98f80781f-service-ca-bundle\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:15.639850 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:15.639787    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 17:46:15.639850 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:15.639797    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15f610bc-3cc9-46de-b61c-bae98f80781f-service-ca-bundle podName:15f610bc-3cc9-46de-b61c-bae98f80781f nodeName:}" failed. No retries permitted until 2026-09-11 17:46:16.639782175 +0000 UTC m=+82.084882738 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/15f610bc-3cc9-46de-b61c-bae98f80781f-service-ca-bundle") pod "router-default-68f69558bb-s4pcz" (UID: "15f610bc-3cc9-46de-b61c-bae98f80781f") : configmap references non-existent config key: service-ca.crt
Sep 11 17:46:15.639992 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:15.639870    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15f610bc-3cc9-46de-b61c-bae98f80781f-metrics-certs podName:15f610bc-3cc9-46de-b61c-bae98f80781f nodeName:}" failed. No retries permitted until 2026-09-11 17:46:16.639852021 +0000 UTC m=+82.084952571 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15f610bc-3cc9-46de-b61c-bae98f80781f-metrics-certs") pod "router-default-68f69558bb-s4pcz" (UID: "15f610bc-3cc9-46de-b61c-bae98f80781f") : secret "router-metrics-certs-default" not found
Sep 11 17:46:15.783945 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.783860    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4hw4h"]
Sep 11 17:46:15.786814 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:15.786787    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-5vb7x"]
Sep 11 17:46:15.787671 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:46:15.787644    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe4e709_ba73_4a3a_9f8c_4e6b68bf25f2.slice/crio-046cd67b1e00ce55101cdd32e2543a69b142cb718f9303d8abf0da8d80e7951f WatchSource:0}: Error finding container 046cd67b1e00ce55101cdd32e2543a69b142cb718f9303d8abf0da8d80e7951f: Status 404 returned error can't find the container with id 046cd67b1e00ce55101cdd32e2543a69b142cb718f9303d8abf0da8d80e7951f
Sep 11 17:46:16.401543 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:16.401499    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5vb7x" event={"ID":"285adb5b-133b-49bd-a8ff-aaf67c7a3cc6","Type":"ContainerStarted","Data":"7733fec34f0f0825a0dc0545675c05a0a44886cf51854abee01c1ba799eb2637"}
Sep 11 17:46:16.402945 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:16.402910    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4hw4h" event={"ID":"abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2","Type":"ContainerStarted","Data":"046cd67b1e00ce55101cdd32e2543a69b142cb718f9303d8abf0da8d80e7951f"}
Sep 11 17:46:16.404163 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:16.404129    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-lcltf" event={"ID":"c539f9b8-d329-47d0-a8ca-79ad211a80c3","Type":"ContainerStarted","Data":"bbb4cc7ab29ffef8be83227abb8de66fa5ae2cf636163a228e2e6d5c769b0005"}
Sep 11 17:46:16.648149 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:16.648091    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15f610bc-3cc9-46de-b61c-bae98f80781f-metrics-certs\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:16.648369 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:16.648162    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f610bc-3cc9-46de-b61c-bae98f80781f-service-ca-bundle\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:16.648369 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:16.648246    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 17:46:16.648369 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:16.648312    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15f610bc-3cc9-46de-b61c-bae98f80781f-service-ca-bundle podName:15f610bc-3cc9-46de-b61c-bae98f80781f nodeName:}" failed. No retries permitted until 2026-09-11 17:46:18.648282627 +0000 UTC m=+84.093383202 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/15f610bc-3cc9-46de-b61c-bae98f80781f-service-ca-bundle") pod "router-default-68f69558bb-s4pcz" (UID: "15f610bc-3cc9-46de-b61c-bae98f80781f") : configmap references non-existent config key: service-ca.crt
Sep 11 17:46:16.648369 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:16.648336    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15f610bc-3cc9-46de-b61c-bae98f80781f-metrics-certs podName:15f610bc-3cc9-46de-b61c-bae98f80781f nodeName:}" failed. No retries permitted until 2026-09-11 17:46:18.64832566 +0000 UTC m=+84.093426224 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15f610bc-3cc9-46de-b61c-bae98f80781f-metrics-certs") pod "router-default-68f69558bb-s4pcz" (UID: "15f610bc-3cc9-46de-b61c-bae98f80781f") : secret "router-metrics-certs-default" not found
Sep 11 17:46:17.284305 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:17.284182    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-d85tk"]
Sep 11 17:46:17.288247 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:17.288225    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-d85tk"
Sep 11 17:46:17.292139 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:17.292112    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-bgc8p\""
Sep 11 17:46:17.299744 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:17.299720    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-d85tk"]
Sep 11 17:46:17.352876 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:17.352833    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t6pj\" (UniqueName: \"kubernetes.io/projected/96d3553b-b0a6-4ebc-b192-4cbaf5ba6766-kube-api-access-7t6pj\") pod \"network-check-source-6f8b768455-d85tk\" (UID: \"96d3553b-b0a6-4ebc-b192-4cbaf5ba6766\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-d85tk"
Sep 11 17:46:17.454149 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:17.454110    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t6pj\" (UniqueName: \"kubernetes.io/projected/96d3553b-b0a6-4ebc-b192-4cbaf5ba6766-kube-api-access-7t6pj\") pod \"network-check-source-6f8b768455-d85tk\" (UID: \"96d3553b-b0a6-4ebc-b192-4cbaf5ba6766\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-d85tk"
Sep 11 17:46:17.469131 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:17.469069    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t6pj\" (UniqueName: \"kubernetes.io/projected/96d3553b-b0a6-4ebc-b192-4cbaf5ba6766-kube-api-access-7t6pj\") pod \"network-check-source-6f8b768455-d85tk\" (UID: \"96d3553b-b0a6-4ebc-b192-4cbaf5ba6766\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-d85tk"
Sep 11 17:46:17.600112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:17.600011    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-d85tk"
Sep 11 17:46:18.664038 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:18.664015    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f610bc-3cc9-46de-b61c-bae98f80781f-service-ca-bundle\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:18.664387 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:18.664095    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15f610bc-3cc9-46de-b61c-bae98f80781f-metrics-certs\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:18.664387 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:18.664182    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15f610bc-3cc9-46de-b61c-bae98f80781f-service-ca-bundle podName:15f610bc-3cc9-46de-b61c-bae98f80781f nodeName:}" failed. No retries permitted until 2026-09-11 17:46:22.664155055 +0000 UTC m=+88.109255623 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/15f610bc-3cc9-46de-b61c-bae98f80781f-service-ca-bundle") pod "router-default-68f69558bb-s4pcz" (UID: "15f610bc-3cc9-46de-b61c-bae98f80781f") : configmap references non-existent config key: service-ca.crt
Sep 11 17:46:18.664387 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:18.664193    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 17:46:18.664387 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:18.664243    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15f610bc-3cc9-46de-b61c-bae98f80781f-metrics-certs podName:15f610bc-3cc9-46de-b61c-bae98f80781f nodeName:}" failed. No retries permitted until 2026-09-11 17:46:22.66422906 +0000 UTC m=+88.109329610 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15f610bc-3cc9-46de-b61c-bae98f80781f-metrics-certs") pod "router-default-68f69558bb-s4pcz" (UID: "15f610bc-3cc9-46de-b61c-bae98f80781f") : secret "router-metrics-certs-default" not found
Sep 11 17:46:18.959733 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:18.959647    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-d85tk"]
Sep 11 17:46:18.966339 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:46:18.966308    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d3553b_b0a6_4ebc_b192_4cbaf5ba6766.slice/crio-7d8cde19e7920f568ac123fd147a306c46c5d9e1e775f56eaa2f99df6937cb67 WatchSource:0}: Error finding container 7d8cde19e7920f568ac123fd147a306c46c5d9e1e775f56eaa2f99df6937cb67: Status 404 returned error can't find the container with id 7d8cde19e7920f568ac123fd147a306c46c5d9e1e775f56eaa2f99df6937cb67
Sep 11 17:46:19.415690 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:19.415655    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-d85tk" event={"ID":"96d3553b-b0a6-4ebc-b192-4cbaf5ba6766","Type":"ContainerStarted","Data":"fc8d98884f8240275e72567578c0c9e2f24aa4bc75a4278868a6e8cf5b1d1aa0"}
Sep 11 17:46:19.416159 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:19.416135    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-d85tk" event={"ID":"96d3553b-b0a6-4ebc-b192-4cbaf5ba6766","Type":"ContainerStarted","Data":"7d8cde19e7920f568ac123fd147a306c46c5d9e1e775f56eaa2f99df6937cb67"}
Sep 11 17:46:19.417551 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:19.417529    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5vb7x" event={"ID":"285adb5b-133b-49bd-a8ff-aaf67c7a3cc6","Type":"ContainerStarted","Data":"6f75b839817fbff74341f2789e4f9cf7f6d84fa15674ef4842795127ef07e7b0"}
Sep 11 17:46:19.419514 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:19.419493    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4hw4h" event={"ID":"abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2","Type":"ContainerStarted","Data":"d198dc14c829b8c17a8c6d0d607c7c968a95420f98ebd3473a67408096a32bc8"}
Sep 11 17:46:19.421644 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:19.421623    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-lcltf" event={"ID":"c539f9b8-d329-47d0-a8ca-79ad211a80c3","Type":"ContainerStarted","Data":"f01e106d3fadcddf03d20894db4f12dbc75f5f5abcd815b20effb0b922d602a6"}
Sep 11 17:46:19.435192 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:19.435034    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-d85tk" podStartSLOduration=2.4350150729999998 podStartE2EDuration="2.435015073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:46:19.433575876 +0000 UTC m=+84.878676445" watchObservedRunningTime="2026-09-11 17:46:19.435015073 +0000 UTC m=+84.880115646"
Sep 11 17:46:19.456466 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:19.456409    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4hw4h" podStartSLOduration=1.555277508 podStartE2EDuration="4.456389305s" totalImagesPullingTime="2.901111797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:46:19.455714641 +0000 UTC m=+84.900815214" watchObservedRunningTime="2026-09-11 17:46:19.456389305 +0000 UTC m=+84.901489871"
Sep 11 17:46:19.498069 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:19.498008    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5vb7x" podStartSLOduration=1.596304478 podStartE2EDuration="4.497988727s" totalImagesPullingTime="2.901684249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:46:19.496937103 +0000 UTC m=+84.942037678" watchObservedRunningTime="2026-09-11 17:46:19.497988727 +0000 UTC m=+84.943089299"
Sep 11 17:46:19.545845 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:19.545794    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-lcltf" podStartSLOduration=2.432779482 podStartE2EDuration="5.545778803s" totalImagesPullingTime="3.112999321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:46:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:46:19.545160909 +0000 UTC m=+84.990261479" watchObservedRunningTime="2026-09-11 17:46:19.545778803 +0000 UTC m=+84.990879373"
Sep 11 17:46:20.438758 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:20.438717    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qx9t7"]
Sep 11 17:46:20.443053 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:20.443028    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:20.447742 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:20.447716    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 17:46:20.447891 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:20.447760    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 17:46:20.449158 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:20.449138    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2rwzx\""
Sep 11 17:46:20.457273 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:20.457247    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qx9t7"]
Sep 11 17:46:20.479709 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:20.479677    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/950b3a2d-acab-436e-85c1-2ebf115f516b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:20.479868 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:20.479792    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqbjx\" (UniqueName: \"kubernetes.io/projected/950b3a2d-acab-436e-85c1-2ebf115f516b-kube-api-access-fqbjx\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:20.479974 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:20.479950    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/950b3a2d-acab-436e-85c1-2ebf115f516b-crio-socket\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:20.480148 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:20.480118    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/950b3a2d-acab-436e-85c1-2ebf115f516b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:20.480224 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:20.480156    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/950b3a2d-acab-436e-85c1-2ebf115f516b-data-volume\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:20.581389 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:20.581327    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/950b3a2d-acab-436e-85c1-2ebf115f516b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:20.581579 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:20.581399    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/950b3a2d-acab-436e-85c1-2ebf115f516b-data-volume\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:20.581579 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:20.581429    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/950b3a2d-acab-436e-85c1-2ebf115f516b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:20.581579 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:20.581470    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:46:20.581579 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:20.581536    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:46:20.581579 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:20.581474    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqbjx\" (UniqueName: \"kubernetes.io/projected/950b3a2d-acab-436e-85c1-2ebf115f516b-kube-api-access-fqbjx\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:20.581823 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:20.581543    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/950b3a2d-acab-436e-85c1-2ebf115f516b-insights-runtime-extractor-tls podName:950b3a2d-acab-436e-85c1-2ebf115f516b nodeName:}" failed. No retries permitted until 2026-09-11 17:46:21.081522642 +0000 UTC m=+86.526623191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/950b3a2d-acab-436e-85c1-2ebf115f516b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qx9t7" (UID: "950b3a2d-acab-436e-85c1-2ebf115f516b") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:46:20.581823 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:20.581616    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/950b3a2d-acab-436e-85c1-2ebf115f516b-kube-rbac-proxy-cm podName:950b3a2d-acab-436e-85c1-2ebf115f516b nodeName:}" failed. No retries permitted until 2026-09-11 17:46:21.081600732 +0000 UTC m=+86.526701280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/950b3a2d-acab-436e-85c1-2ebf115f516b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qx9t7" (UID: "950b3a2d-acab-436e-85c1-2ebf115f516b") : configmap "kube-rbac-proxy" not found
Sep 11 17:46:20.581823 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:20.581689    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/950b3a2d-acab-436e-85c1-2ebf115f516b-crio-socket\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:20.581823 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:20.581806    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/950b3a2d-acab-436e-85c1-2ebf115f516b-data-volume\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:20.581974 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:20.581812    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/950b3a2d-acab-436e-85c1-2ebf115f516b-crio-socket\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:20.614201 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:20.614171    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqbjx\" (UniqueName: \"kubernetes.io/projected/950b3a2d-acab-436e-85c1-2ebf115f516b-kube-api-access-fqbjx\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:21.085250 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:21.085193    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/950b3a2d-acab-436e-85c1-2ebf115f516b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:21.085464 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:21.085316    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:46:21.085464 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:21.085378    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/950b3a2d-acab-436e-85c1-2ebf115f516b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:21.085464 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:21.085407    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/950b3a2d-acab-436e-85c1-2ebf115f516b-insights-runtime-extractor-tls podName:950b3a2d-acab-436e-85c1-2ebf115f516b nodeName:}" failed. No retries permitted until 2026-09-11 17:46:22.085385043 +0000 UTC m=+87.530485603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/950b3a2d-acab-436e-85c1-2ebf115f516b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qx9t7" (UID: "950b3a2d-acab-436e-85c1-2ebf115f516b") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:46:21.085464 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:21.085437    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:46:21.085636 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:21.085487    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/950b3a2d-acab-436e-85c1-2ebf115f516b-kube-rbac-proxy-cm podName:950b3a2d-acab-436e-85c1-2ebf115f516b nodeName:}" failed. No retries permitted until 2026-09-11 17:46:22.085473086 +0000 UTC m=+87.530573665 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/950b3a2d-acab-436e-85c1-2ebf115f516b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qx9t7" (UID: "950b3a2d-acab-436e-85c1-2ebf115f516b") : configmap "kube-rbac-proxy" not found
Sep 11 17:46:22.094153 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.094101    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/950b3a2d-acab-436e-85c1-2ebf115f516b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:22.094581 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.094164    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/950b3a2d-acab-436e-85c1-2ebf115f516b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:22.094581 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:22.094230    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:46:22.094581 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:22.094247    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:46:22.094581 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:22.094312    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/950b3a2d-acab-436e-85c1-2ebf115f516b-kube-rbac-proxy-cm podName:950b3a2d-acab-436e-85c1-2ebf115f516b nodeName:}" failed. No retries permitted until 2026-09-11 17:46:24.094291951 +0000 UTC m=+89.539392523 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/950b3a2d-acab-436e-85c1-2ebf115f516b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qx9t7" (UID: "950b3a2d-acab-436e-85c1-2ebf115f516b") : configmap "kube-rbac-proxy" not found
Sep 11 17:46:22.094581 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:22.094333    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/950b3a2d-acab-436e-85c1-2ebf115f516b-insights-runtime-extractor-tls podName:950b3a2d-acab-436e-85c1-2ebf115f516b nodeName:}" failed. No retries permitted until 2026-09-11 17:46:24.094323884 +0000 UTC m=+89.539424433 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/950b3a2d-acab-436e-85c1-2ebf115f516b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qx9t7" (UID: "950b3a2d-acab-436e-85c1-2ebf115f516b") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:46:22.183102 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.183074    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5qftl_190c54c5-ae20-4ef1-b9b2-af966c1712f4/dns-node-resolver/0.log"
Sep 11 17:46:22.292918 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.292881    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-j4nxr"]
Sep 11 17:46:22.296043 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.296025    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j4nxr"
Sep 11 17:46:22.303555 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.303534    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 11 17:46:22.303555 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.303549    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 11 17:46:22.303839 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.303822    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-f9sxt\""
Sep 11 17:46:22.320244 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.320213    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-j4nxr"]
Sep 11 17:46:22.396601 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.396568    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca7f28e-65fb-42e3-825a-f3100e244207-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j4nxr\" (UID: \"3ca7f28e-65fb-42e3-825a-f3100e244207\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j4nxr"
Sep 11 17:46:22.396601 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.396607    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3ca7f28e-65fb-42e3-825a-f3100e244207-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-j4nxr\" (UID: \"3ca7f28e-65fb-42e3-825a-f3100e244207\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j4nxr"
Sep 11 17:46:22.497863 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.497826    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca7f28e-65fb-42e3-825a-f3100e244207-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j4nxr\" (UID: \"3ca7f28e-65fb-42e3-825a-f3100e244207\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j4nxr"
Sep 11 17:46:22.497863 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.497864    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3ca7f28e-65fb-42e3-825a-f3100e244207-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-j4nxr\" (UID: \"3ca7f28e-65fb-42e3-825a-f3100e244207\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j4nxr"
Sep 11 17:46:22.498062 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:22.497994    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 17:46:22.498096 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:22.498078    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca7f28e-65fb-42e3-825a-f3100e244207-networking-console-plugin-cert podName:3ca7f28e-65fb-42e3-825a-f3100e244207 nodeName:}" failed. No retries permitted until 2026-09-11 17:46:22.998061802 +0000 UTC m=+88.443162364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ca7f28e-65fb-42e3-825a-f3100e244207-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-j4nxr" (UID: "3ca7f28e-65fb-42e3-825a-f3100e244207") : secret "networking-console-plugin-cert" not found
Sep 11 17:46:22.498533 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.498516    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3ca7f28e-65fb-42e3-825a-f3100e244207-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-j4nxr\" (UID: \"3ca7f28e-65fb-42e3-825a-f3100e244207\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j4nxr"
Sep 11 17:46:22.546795 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.546757    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-qgjbb"]
Sep 11 17:46:22.549973 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.549954    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-qgjbb"
Sep 11 17:46:22.554450 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.554425    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 11 17:46:22.554710 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.554691    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 11 17:46:22.554806 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.554788    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5srcr\""
Sep 11 17:46:22.554888 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.554872    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 11 17:46:22.555157 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.555126    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 11 17:46:22.570794 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.570769    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-qgjbb"]
Sep 11 17:46:22.599068 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.599037    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/01a9996b-c602-459a-a91f-e11581e14293-signing-key\") pod \"service-ca-568869fdc5-qgjbb\" (UID: \"01a9996b-c602-459a-a91f-e11581e14293\") " pod="openshift-service-ca/service-ca-568869fdc5-qgjbb"
Sep 11 17:46:22.599240 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.599117    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/01a9996b-c602-459a-a91f-e11581e14293-signing-cabundle\") pod \"service-ca-568869fdc5-qgjbb\" (UID: \"01a9996b-c602-459a-a91f-e11581e14293\") " pod="openshift-service-ca/service-ca-568869fdc5-qgjbb"
Sep 11 17:46:22.599240 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.599136    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pwjd\" (UniqueName: \"kubernetes.io/projected/01a9996b-c602-459a-a91f-e11581e14293-kube-api-access-6pwjd\") pod \"service-ca-568869fdc5-qgjbb\" (UID: \"01a9996b-c602-459a-a91f-e11581e14293\") " pod="openshift-service-ca/service-ca-568869fdc5-qgjbb"
Sep 11 17:46:22.699702 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.699612    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/01a9996b-c602-459a-a91f-e11581e14293-signing-key\") pod \"service-ca-568869fdc5-qgjbb\" (UID: \"01a9996b-c602-459a-a91f-e11581e14293\") " pod="openshift-service-ca/service-ca-568869fdc5-qgjbb"
Sep 11 17:46:22.699702 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.699670    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f610bc-3cc9-46de-b61c-bae98f80781f-service-ca-bundle\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:22.699702 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.699686    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15f610bc-3cc9-46de-b61c-bae98f80781f-metrics-certs\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:22.699702 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.699704    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/01a9996b-c602-459a-a91f-e11581e14293-signing-cabundle\") pod \"service-ca-568869fdc5-qgjbb\" (UID: \"01a9996b-c602-459a-a91f-e11581e14293\") " pod="openshift-service-ca/service-ca-568869fdc5-qgjbb"
Sep 11 17:46:22.700026 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.699728    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pwjd\" (UniqueName: \"kubernetes.io/projected/01a9996b-c602-459a-a91f-e11581e14293-kube-api-access-6pwjd\") pod \"service-ca-568869fdc5-qgjbb\" (UID: \"01a9996b-c602-459a-a91f-e11581e14293\") " pod="openshift-service-ca/service-ca-568869fdc5-qgjbb"
Sep 11 17:46:22.700026 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:22.699797    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 17:46:22.700026 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:22.699811    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15f610bc-3cc9-46de-b61c-bae98f80781f-service-ca-bundle podName:15f610bc-3cc9-46de-b61c-bae98f80781f nodeName:}" failed. No retries permitted until 2026-09-11 17:46:30.699793873 +0000 UTC m=+96.144894444 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/15f610bc-3cc9-46de-b61c-bae98f80781f-service-ca-bundle") pod "router-default-68f69558bb-s4pcz" (UID: "15f610bc-3cc9-46de-b61c-bae98f80781f") : configmap references non-existent config key: service-ca.crt
Sep 11 17:46:22.700026 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:22.699875    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15f610bc-3cc9-46de-b61c-bae98f80781f-metrics-certs podName:15f610bc-3cc9-46de-b61c-bae98f80781f nodeName:}" failed. No retries permitted until 2026-09-11 17:46:30.69985982 +0000 UTC m=+96.144960369 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15f610bc-3cc9-46de-b61c-bae98f80781f-metrics-certs") pod "router-default-68f69558bb-s4pcz" (UID: "15f610bc-3cc9-46de-b61c-bae98f80781f") : secret "router-metrics-certs-default" not found
Sep 11 17:46:22.700480 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.700461    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/01a9996b-c602-459a-a91f-e11581e14293-signing-cabundle\") pod \"service-ca-568869fdc5-qgjbb\" (UID: \"01a9996b-c602-459a-a91f-e11581e14293\") " pod="openshift-service-ca/service-ca-568869fdc5-qgjbb"
Sep 11 17:46:22.702126 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.702102    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/01a9996b-c602-459a-a91f-e11581e14293-signing-key\") pod \"service-ca-568869fdc5-qgjbb\" (UID: \"01a9996b-c602-459a-a91f-e11581e14293\") " pod="openshift-service-ca/service-ca-568869fdc5-qgjbb"
Sep 11 17:46:22.715198 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.715176    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pwjd\" (UniqueName: \"kubernetes.io/projected/01a9996b-c602-459a-a91f-e11581e14293-kube-api-access-6pwjd\") pod \"service-ca-568869fdc5-qgjbb\" (UID: \"01a9996b-c602-459a-a91f-e11581e14293\") " pod="openshift-service-ca/service-ca-568869fdc5-qgjbb"
Sep 11 17:46:22.859024 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:22.858980    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-qgjbb"
Sep 11 17:46:23.002264 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:23.002181    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca7f28e-65fb-42e3-825a-f3100e244207-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j4nxr\" (UID: \"3ca7f28e-65fb-42e3-825a-f3100e244207\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j4nxr"
Sep 11 17:46:23.002460 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:23.002317    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 17:46:23.002460 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:23.002398    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca7f28e-65fb-42e3-825a-f3100e244207-networking-console-plugin-cert podName:3ca7f28e-65fb-42e3-825a-f3100e244207 nodeName:}" failed. No retries permitted until 2026-09-11 17:46:24.002378853 +0000 UTC m=+89.447479425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ca7f28e-65fb-42e3-825a-f3100e244207-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-j4nxr" (UID: "3ca7f28e-65fb-42e3-825a-f3100e244207") : secret "networking-console-plugin-cert" not found
Sep 11 17:46:23.141692 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:23.141660    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-qgjbb"]
Sep 11 17:46:23.145160 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:46:23.145125    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a9996b_c602_459a_a91f_e11581e14293.slice/crio-b97a239a98231effb2b87f6a123186ed39c0402a21b20a806d2cecd849506e8d WatchSource:0}: Error finding container b97a239a98231effb2b87f6a123186ed39c0402a21b20a806d2cecd849506e8d: Status 404 returned error can't find the container with id b97a239a98231effb2b87f6a123186ed39c0402a21b20a806d2cecd849506e8d
Sep 11 17:46:23.375517 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:23.375438    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kpjjv_a8238ce9-9f28-4900-8b41-667fa71978f2/node-ca/0.log"
Sep 11 17:46:23.436394 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:23.436344    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-qgjbb" event={"ID":"01a9996b-c602-459a-a91f-e11581e14293","Type":"ContainerStarted","Data":"782311cacabcea8a83844cf203db2b72df40630fdf5c0e2938d098cdc695c9c1"}
Sep 11 17:46:23.436585 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:23.436408    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-qgjbb" event={"ID":"01a9996b-c602-459a-a91f-e11581e14293","Type":"ContainerStarted","Data":"b97a239a98231effb2b87f6a123186ed39c0402a21b20a806d2cecd849506e8d"}
Sep 11 17:46:23.464055 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:23.463997    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-qgjbb" podStartSLOduration=1.463979887 podStartE2EDuration="1.463979887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:46:23.463508699 +0000 UTC m=+88.908609270" watchObservedRunningTime="2026-09-11 17:46:23.463979887 +0000 UTC m=+88.909080458"
Sep 11 17:46:23.976485 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:23.976450    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cspnd_37764305-549b-498a-967e-c24cbc84e019/serve-healthcheck-canary/0.log"
Sep 11 17:46:24.012929 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:24.012895    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca7f28e-65fb-42e3-825a-f3100e244207-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j4nxr\" (UID: \"3ca7f28e-65fb-42e3-825a-f3100e244207\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j4nxr"
Sep 11 17:46:24.013053 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:24.013037    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 17:46:24.013125 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:24.013116    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca7f28e-65fb-42e3-825a-f3100e244207-networking-console-plugin-cert podName:3ca7f28e-65fb-42e3-825a-f3100e244207 nodeName:}" failed. No retries permitted until 2026-09-11 17:46:26.013097184 +0000 UTC m=+91.458197752 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ca7f28e-65fb-42e3-825a-f3100e244207-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-j4nxr" (UID: "3ca7f28e-65fb-42e3-825a-f3100e244207") : secret "networking-console-plugin-cert" not found
Sep 11 17:46:24.114038 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:24.113989    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/950b3a2d-acab-436e-85c1-2ebf115f516b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:24.114038 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:24.114043    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/950b3a2d-acab-436e-85c1-2ebf115f516b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:24.114292 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:24.114144    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:46:24.114292 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:24.114186    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:46:24.114292 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:24.114212    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/950b3a2d-acab-436e-85c1-2ebf115f516b-insights-runtime-extractor-tls podName:950b3a2d-acab-436e-85c1-2ebf115f516b nodeName:}" failed. No retries permitted until 2026-09-11 17:46:28.114196401 +0000 UTC m=+93.559296965 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/950b3a2d-acab-436e-85c1-2ebf115f516b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qx9t7" (UID: "950b3a2d-acab-436e-85c1-2ebf115f516b") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:46:24.114292 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:24.114250    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/950b3a2d-acab-436e-85c1-2ebf115f516b-kube-rbac-proxy-cm podName:950b3a2d-acab-436e-85c1-2ebf115f516b nodeName:}" failed. No retries permitted until 2026-09-11 17:46:28.114231039 +0000 UTC m=+93.559331598 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/950b3a2d-acab-436e-85c1-2ebf115f516b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qx9t7" (UID: "950b3a2d-acab-436e-85c1-2ebf115f516b") : configmap "kube-rbac-proxy" not found
Sep 11 17:46:26.031770 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:26.031730    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca7f28e-65fb-42e3-825a-f3100e244207-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j4nxr\" (UID: \"3ca7f28e-65fb-42e3-825a-f3100e244207\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j4nxr"
Sep 11 17:46:26.032201 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:26.031877    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 17:46:26.032201 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:26.031947    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca7f28e-65fb-42e3-825a-f3100e244207-networking-console-plugin-cert podName:3ca7f28e-65fb-42e3-825a-f3100e244207 nodeName:}" failed. No retries permitted until 2026-09-11 17:46:30.031931154 +0000 UTC m=+95.477031703 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ca7f28e-65fb-42e3-825a-f3100e244207-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-j4nxr" (UID: "3ca7f28e-65fb-42e3-825a-f3100e244207") : secret "networking-console-plugin-cert" not found
Sep 11 17:46:28.148096 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:28.148034    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/950b3a2d-acab-436e-85c1-2ebf115f516b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:28.148096 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:28.148103    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/950b3a2d-acab-436e-85c1-2ebf115f516b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:28.148616 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:28.148197    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:46:28.148616 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:28.148237    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:46:28.148616 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:28.148266    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/950b3a2d-acab-436e-85c1-2ebf115f516b-insights-runtime-extractor-tls podName:950b3a2d-acab-436e-85c1-2ebf115f516b nodeName:}" failed. No retries permitted until 2026-09-11 17:46:36.148250917 +0000 UTC m=+101.593351465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/950b3a2d-acab-436e-85c1-2ebf115f516b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qx9t7" (UID: "950b3a2d-acab-436e-85c1-2ebf115f516b") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:46:28.148616 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:28.148289    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/950b3a2d-acab-436e-85c1-2ebf115f516b-kube-rbac-proxy-cm podName:950b3a2d-acab-436e-85c1-2ebf115f516b nodeName:}" failed. No retries permitted until 2026-09-11 17:46:36.148276883 +0000 UTC m=+101.593377432 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/950b3a2d-acab-436e-85c1-2ebf115f516b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qx9t7" (UID: "950b3a2d-acab-436e-85c1-2ebf115f516b") : configmap "kube-rbac-proxy" not found
Sep 11 17:46:30.063780 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:30.063741    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca7f28e-65fb-42e3-825a-f3100e244207-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j4nxr\" (UID: \"3ca7f28e-65fb-42e3-825a-f3100e244207\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j4nxr"
Sep 11 17:46:30.064233 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:30.063891    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 17:46:30.064233 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:46:30.063961    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca7f28e-65fb-42e3-825a-f3100e244207-networking-console-plugin-cert podName:3ca7f28e-65fb-42e3-825a-f3100e244207 nodeName:}" failed. No retries permitted until 2026-09-11 17:46:38.06394577 +0000 UTC m=+103.509046319 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ca7f28e-65fb-42e3-825a-f3100e244207-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-j4nxr" (UID: "3ca7f28e-65fb-42e3-825a-f3100e244207") : secret "networking-console-plugin-cert" not found
Sep 11 17:46:30.771494 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:30.771457    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f610bc-3cc9-46de-b61c-bae98f80781f-service-ca-bundle\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:30.771494 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:30.771507    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15f610bc-3cc9-46de-b61c-bae98f80781f-metrics-certs\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:30.772162 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:30.772130    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f610bc-3cc9-46de-b61c-bae98f80781f-service-ca-bundle\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:30.774080 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:30.774057    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15f610bc-3cc9-46de-b61c-bae98f80781f-metrics-certs\") pod \"router-default-68f69558bb-s4pcz\" (UID: \"15f610bc-3cc9-46de-b61c-bae98f80781f\") " pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:30.902929 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:30.902871    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:31.218077 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:31.218049    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-68f69558bb-s4pcz"]
Sep 11 17:46:31.220967 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:46:31.220937    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f610bc_3cc9_46de_b61c_bae98f80781f.slice/crio-394585d2e38a47bf0bf6b13910169ce58a5624b3a5ec85d18e3a1bf9cfd1afb2 WatchSource:0}: Error finding container 394585d2e38a47bf0bf6b13910169ce58a5624b3a5ec85d18e3a1bf9cfd1afb2: Status 404 returned error can't find the container with id 394585d2e38a47bf0bf6b13910169ce58a5624b3a5ec85d18e3a1bf9cfd1afb2
Sep 11 17:46:31.470309 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:31.470223    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-68f69558bb-s4pcz" event={"ID":"15f610bc-3cc9-46de-b61c-bae98f80781f","Type":"ContainerStarted","Data":"d7569179c7d93e6413173f4959918eed375470e353aa33b7b938b90651985092"}
Sep 11 17:46:31.470309 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:31.470300    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-68f69558bb-s4pcz" event={"ID":"15f610bc-3cc9-46de-b61c-bae98f80781f","Type":"ContainerStarted","Data":"394585d2e38a47bf0bf6b13910169ce58a5624b3a5ec85d18e3a1bf9cfd1afb2"}
Sep 11 17:46:31.903834 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:31.903796    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:31.906296 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:31.906270    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:31.945454 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:31.945402    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-68f69558bb-s4pcz" podStartSLOduration=17.945384161 podStartE2EDuration="17.945384161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:46:31.506149074 +0000 UTC m=+96.951249666" watchObservedRunningTime="2026-09-11 17:46:31.945384161 +0000 UTC m=+97.390484731"
Sep 11 17:46:32.473685 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:32.473652    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:32.474974 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:32.474951    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-68f69558bb-s4pcz"
Sep 11 17:46:34.300223 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:34.300181    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e831333a-4c9b-4a40-a53e-c66f3e2f653f-metrics-tls\") pod \"dns-default-h8grx\" (UID: \"e831333a-4c9b-4a40-a53e-c66f3e2f653f\") " pod="openshift-dns/dns-default-h8grx"
Sep 11 17:46:34.302482 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:34.302461    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e831333a-4c9b-4a40-a53e-c66f3e2f653f-metrics-tls\") pod \"dns-default-h8grx\" (UID: \"e831333a-4c9b-4a40-a53e-c66f3e2f653f\") " pod="openshift-dns/dns-default-h8grx"
Sep 11 17:46:34.421764 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:34.421733    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tc7ww\""
Sep 11 17:46:34.428303 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:34.428282    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h8grx"
Sep 11 17:46:34.713505 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:34.713470    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h8grx"]
Sep 11 17:46:34.716227 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:46:34.716199    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode831333a_4c9b_4a40_a53e_c66f3e2f653f.slice/crio-c4cb0aa7abd8085b6b822b55b04097355da783c6b11f223455ee135a260b5331 WatchSource:0}: Error finding container c4cb0aa7abd8085b6b822b55b04097355da783c6b11f223455ee135a260b5331: Status 404 returned error can't find the container with id c4cb0aa7abd8085b6b822b55b04097355da783c6b11f223455ee135a260b5331
Sep 11 17:46:35.485608 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:35.485576    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h8grx" event={"ID":"e831333a-4c9b-4a40-a53e-c66f3e2f653f","Type":"ContainerStarted","Data":"c4cb0aa7abd8085b6b822b55b04097355da783c6b11f223455ee135a260b5331"}
Sep 11 17:46:36.217582 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:36.217543    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/950b3a2d-acab-436e-85c1-2ebf115f516b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:36.217582 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:36.217583    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/950b3a2d-acab-436e-85c1-2ebf115f516b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:36.219372 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:36.218731    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/950b3a2d-acab-436e-85c1-2ebf115f516b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:36.223245 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:36.223219    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/950b3a2d-acab-436e-85c1-2ebf115f516b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qx9t7\" (UID: \"950b3a2d-acab-436e-85c1-2ebf115f516b\") " pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:36.353153 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:36.353109    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qx9t7"
Sep 11 17:46:36.727309 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:36.727275    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qx9t7"]
Sep 11 17:46:36.731577 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:46:36.731546    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950b3a2d_acab_436e_85c1_2ebf115f516b.slice/crio-304ac4bd890d69c7910a1ef7b4234765707893de160ee00cf19602c86736edb1 WatchSource:0}: Error finding container 304ac4bd890d69c7910a1ef7b4234765707893de160ee00cf19602c86736edb1: Status 404 returned error can't find the container with id 304ac4bd890d69c7910a1ef7b4234765707893de160ee00cf19602c86736edb1
Sep 11 17:46:37.382456 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:37.382428    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8nchm"
Sep 11 17:46:37.494973 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:37.494937    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h8grx" event={"ID":"e831333a-4c9b-4a40-a53e-c66f3e2f653f","Type":"ContainerStarted","Data":"ecf66bd8925d49cea04bad5af109859e955cbb59c213aba917f2bff4fe354d25"}
Sep 11 17:46:37.495155 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:37.495082    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-h8grx"
Sep 11 17:46:37.495155 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:37.495120    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h8grx" event={"ID":"e831333a-4c9b-4a40-a53e-c66f3e2f653f","Type":"ContainerStarted","Data":"fbecf2927f8d6401d2dcb795db4b6f8598eaa26d14d935c7f02595bd0a0e4448"}
Sep 11 17:46:37.496823 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:37.496799    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qx9t7" event={"ID":"950b3a2d-acab-436e-85c1-2ebf115f516b","Type":"ContainerStarted","Data":"79a4b3166d618345707192e1a4e9b1cde97f6729196867190681cfcc87733579"}
Sep 11 17:46:37.496931 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:37.496843    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qx9t7" event={"ID":"950b3a2d-acab-436e-85c1-2ebf115f516b","Type":"ContainerStarted","Data":"7d1579e708b8d369194b6121d2a8bed81c9e2f802c12ed21f7c67e4ecf36ccd1"}
Sep 11 17:46:37.496931 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:37.496862    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qx9t7" event={"ID":"950b3a2d-acab-436e-85c1-2ebf115f516b","Type":"ContainerStarted","Data":"304ac4bd890d69c7910a1ef7b4234765707893de160ee00cf19602c86736edb1"}
Sep 11 17:46:37.543872 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:37.543816    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-h8grx" podStartSLOduration=65.76078659 podStartE2EDuration="1m7.543800561s" totalImagesPullingTime="1.783013971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:45:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:46:37.54185127 +0000 UTC m=+102.986951840" watchObservedRunningTime="2026-09-11 17:46:37.543800561 +0000 UTC m=+102.988901191"
Sep 11 17:46:38.134241 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:38.134198    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca7f28e-65fb-42e3-825a-f3100e244207-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j4nxr\" (UID: \"3ca7f28e-65fb-42e3-825a-f3100e244207\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j4nxr"
Sep 11 17:46:38.136871 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:38.136841    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ca7f28e-65fb-42e3-825a-f3100e244207-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-j4nxr\" (UID: \"3ca7f28e-65fb-42e3-825a-f3100e244207\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j4nxr"
Sep 11 17:46:38.204480 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:38.204443    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j4nxr"
Sep 11 17:46:38.522049 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:38.522010    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-j4nxr"]
Sep 11 17:46:38.969520 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:46:38.969483    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca7f28e_65fb_42e3_825a_f3100e244207.slice/crio-ba366cd6da6094875e8d6865118ea0a4468b328c1b315c03e9c8783941b51f44 WatchSource:0}: Error finding container ba366cd6da6094875e8d6865118ea0a4468b328c1b315c03e9c8783941b51f44: Status 404 returned error can't find the container with id ba366cd6da6094875e8d6865118ea0a4468b328c1b315c03e9c8783941b51f44
Sep 11 17:46:39.506380 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:39.506328    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qx9t7" event={"ID":"950b3a2d-acab-436e-85c1-2ebf115f516b","Type":"ContainerStarted","Data":"bb4e4e82987241c2812609e1a2c0e9223f5b67271115fe04d254b681fc29d5a2"}
Sep 11 17:46:39.507583 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:39.507560    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j4nxr" event={"ID":"3ca7f28e-65fb-42e3-825a-f3100e244207","Type":"ContainerStarted","Data":"ba366cd6da6094875e8d6865118ea0a4468b328c1b315c03e9c8783941b51f44"}
Sep 11 17:46:41.515139 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:41.515096    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j4nxr" event={"ID":"3ca7f28e-65fb-42e3-825a-f3100e244207","Type":"ContainerStarted","Data":"a359f7c85c22187835f20386781105bf0b01d4fd3f16699e0466972c2d8ae163"}
Sep 11 17:46:41.548185 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:41.548119    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qx9t7" podStartSLOduration=19.082698164 podStartE2EDuration="21.548101562s" totalImagesPullingTime="2.465403398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:46:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:46:39.545113762 +0000 UTC m=+104.990214332" watchObservedRunningTime="2026-09-11 17:46:41.548101562 +0000 UTC m=+106.993202134"
Sep 11 17:46:47.502608 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:47.502575    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h8grx"
Sep 11 17:46:47.543219 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:47.543164    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-j4nxr" podStartSLOduration=23.534806252 podStartE2EDuration="25.543148215s" totalImagesPullingTime="2.008341963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:46:41.548256152 +0000 UTC m=+106.993356707" watchObservedRunningTime="2026-09-11 17:46:47.543148215 +0000 UTC m=+112.988248786"
Sep 11 17:46:51.092080 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.092052    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-5d2qv"]
Sep 11 17:46:51.094832 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.094816    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-5d2qv"
Sep 11 17:46:51.104268 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.104247    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 17:46:51.111009 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.110990    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 17:46:51.111184 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.111167    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-ngrjm\""
Sep 11 17:46:51.150688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.150660    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-5d2qv"]
Sep 11 17:46:51.238775 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.238735    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9tzq\" (UniqueName: \"kubernetes.io/projected/4546e901-4e48-42bd-860c-54b99428070a-kube-api-access-q9tzq\") pod \"downloads-76f99584f4-5d2qv\" (UID: \"4546e901-4e48-42bd-860c-54b99428070a\") " pod="openshift-console/downloads-76f99584f4-5d2qv"
Sep 11 17:46:51.265986 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.265953    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59cc77dfdd-2vlmn"]
Sep 11 17:46:51.268538 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.268519    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.277482 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.277462    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 17:46:51.278433 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.278417    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 17:46:51.278921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.278904    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 17:46:51.279226 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.279209    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-h7qvz\""
Sep 11 17:46:51.285038 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.285014    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59cc77dfdd-2vlmn"]
Sep 11 17:46:51.299775 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.299712    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 17:46:51.339280 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.339245    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9tzq\" (UniqueName: \"kubernetes.io/projected/4546e901-4e48-42bd-860c-54b99428070a-kube-api-access-q9tzq\") pod \"downloads-76f99584f4-5d2qv\" (UID: \"4546e901-4e48-42bd-860c-54b99428070a\") " pod="openshift-console/downloads-76f99584f4-5d2qv"
Sep 11 17:46:51.352577 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.352545    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9tzq\" (UniqueName: \"kubernetes.io/projected/4546e901-4e48-42bd-860c-54b99428070a-kube-api-access-q9tzq\") pod \"downloads-76f99584f4-5d2qv\" (UID: \"4546e901-4e48-42bd-860c-54b99428070a\") " pod="openshift-console/downloads-76f99584f4-5d2qv"
Sep 11 17:46:51.403667 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.403637    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-5d2qv"
Sep 11 17:46:51.440045 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.439996    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d650e928-1422-4436-8122-308bcb12e93b-installation-pull-secrets\") pod \"image-registry-59cc77dfdd-2vlmn\" (UID: \"d650e928-1422-4436-8122-308bcb12e93b\") " pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.440228 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.440054    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w4km\" (UniqueName: \"kubernetes.io/projected/d650e928-1422-4436-8122-308bcb12e93b-kube-api-access-4w4km\") pod \"image-registry-59cc77dfdd-2vlmn\" (UID: \"d650e928-1422-4436-8122-308bcb12e93b\") " pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.440293 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.440250    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d650e928-1422-4436-8122-308bcb12e93b-registry-certificates\") pod \"image-registry-59cc77dfdd-2vlmn\" (UID: \"d650e928-1422-4436-8122-308bcb12e93b\") " pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.440346 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.440288    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d650e928-1422-4436-8122-308bcb12e93b-bound-sa-token\") pod \"image-registry-59cc77dfdd-2vlmn\" (UID: \"d650e928-1422-4436-8122-308bcb12e93b\") " pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.440346 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.440317    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d650e928-1422-4436-8122-308bcb12e93b-registry-tls\") pod \"image-registry-59cc77dfdd-2vlmn\" (UID: \"d650e928-1422-4436-8122-308bcb12e93b\") " pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.440462 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.440341    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d650e928-1422-4436-8122-308bcb12e93b-ca-trust-extracted\") pod \"image-registry-59cc77dfdd-2vlmn\" (UID: \"d650e928-1422-4436-8122-308bcb12e93b\") " pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.440509 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.440461    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d650e928-1422-4436-8122-308bcb12e93b-trusted-ca\") pod \"image-registry-59cc77dfdd-2vlmn\" (UID: \"d650e928-1422-4436-8122-308bcb12e93b\") " pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.440509 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.440497    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d650e928-1422-4436-8122-308bcb12e93b-image-registry-private-configuration\") pod \"image-registry-59cc77dfdd-2vlmn\" (UID: \"d650e928-1422-4436-8122-308bcb12e93b\") " pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.541812 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.541777    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d650e928-1422-4436-8122-308bcb12e93b-installation-pull-secrets\") pod \"image-registry-59cc77dfdd-2vlmn\" (UID: \"d650e928-1422-4436-8122-308bcb12e93b\") " pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.541812 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.541815    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w4km\" (UniqueName: \"kubernetes.io/projected/d650e928-1422-4436-8122-308bcb12e93b-kube-api-access-4w4km\") pod \"image-registry-59cc77dfdd-2vlmn\" (UID: \"d650e928-1422-4436-8122-308bcb12e93b\") " pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.542048 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.542016    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d650e928-1422-4436-8122-308bcb12e93b-registry-certificates\") pod \"image-registry-59cc77dfdd-2vlmn\" (UID: \"d650e928-1422-4436-8122-308bcb12e93b\") " pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.542109 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.542062    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d650e928-1422-4436-8122-308bcb12e93b-bound-sa-token\") pod \"image-registry-59cc77dfdd-2vlmn\" (UID: \"d650e928-1422-4436-8122-308bcb12e93b\") " pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.542109 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.542092    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d650e928-1422-4436-8122-308bcb12e93b-registry-tls\") pod \"image-registry-59cc77dfdd-2vlmn\" (UID: \"d650e928-1422-4436-8122-308bcb12e93b\") " pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.542204 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.542117    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d650e928-1422-4436-8122-308bcb12e93b-ca-trust-extracted\") pod \"image-registry-59cc77dfdd-2vlmn\" (UID: \"d650e928-1422-4436-8122-308bcb12e93b\") " pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.542204 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.542177    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d650e928-1422-4436-8122-308bcb12e93b-trusted-ca\") pod \"image-registry-59cc77dfdd-2vlmn\" (UID: \"d650e928-1422-4436-8122-308bcb12e93b\") " pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.542301 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.542223    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d650e928-1422-4436-8122-308bcb12e93b-image-registry-private-configuration\") pod \"image-registry-59cc77dfdd-2vlmn\" (UID: \"d650e928-1422-4436-8122-308bcb12e93b\") " pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.542666 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.542637    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d650e928-1422-4436-8122-308bcb12e93b-ca-trust-extracted\") pod \"image-registry-59cc77dfdd-2vlmn\" (UID: \"d650e928-1422-4436-8122-308bcb12e93b\") " pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.542936 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.542913    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d650e928-1422-4436-8122-308bcb12e93b-registry-certificates\") pod \"image-registry-59cc77dfdd-2vlmn\" (UID: \"d650e928-1422-4436-8122-308bcb12e93b\") " pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.543095 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.543059    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d650e928-1422-4436-8122-308bcb12e93b-trusted-ca\") pod \"image-registry-59cc77dfdd-2vlmn\" (UID: \"d650e928-1422-4436-8122-308bcb12e93b\") " pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.544669 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.544636    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d650e928-1422-4436-8122-308bcb12e93b-installation-pull-secrets\") pod \"image-registry-59cc77dfdd-2vlmn\" (UID: \"d650e928-1422-4436-8122-308bcb12e93b\") " pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.544902 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.544884    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d650e928-1422-4436-8122-308bcb12e93b-registry-tls\") pod \"image-registry-59cc77dfdd-2vlmn\" (UID: \"d650e928-1422-4436-8122-308bcb12e93b\") " pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.545021 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.545001    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d650e928-1422-4436-8122-308bcb12e93b-image-registry-private-configuration\") pod \"image-registry-59cc77dfdd-2vlmn\" (UID: \"d650e928-1422-4436-8122-308bcb12e93b\") " pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.556461 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.556405    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w4km\" (UniqueName: \"kubernetes.io/projected/d650e928-1422-4436-8122-308bcb12e93b-kube-api-access-4w4km\") pod \"image-registry-59cc77dfdd-2vlmn\" (UID: \"d650e928-1422-4436-8122-308bcb12e93b\") " pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.557313 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.557284    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d650e928-1422-4436-8122-308bcb12e93b-bound-sa-token\") pod \"image-registry-59cc77dfdd-2vlmn\" (UID: \"d650e928-1422-4436-8122-308bcb12e93b\") " pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.577470 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.577444    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:51.689458 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.689426    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-5d2qv"]
Sep 11 17:46:51.692416 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:46:51.692387    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4546e901_4e48_42bd_860c_54b99428070a.slice/crio-35b8fbf54b72f4a0fd545009df5af19d6d72d5d96f89b5cf39e554e746f4d70b WatchSource:0}: Error finding container 35b8fbf54b72f4a0fd545009df5af19d6d72d5d96f89b5cf39e554e746f4d70b: Status 404 returned error can't find the container with id 35b8fbf54b72f4a0fd545009df5af19d6d72d5d96f89b5cf39e554e746f4d70b
Sep 11 17:46:51.880507 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:51.880433    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59cc77dfdd-2vlmn"]
Sep 11 17:46:51.884014 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:46:51.883954    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd650e928_1422_4436_8122_308bcb12e93b.slice/crio-f93ed15b10ca9fb9af6ca874e812e9c229ad30fc3be7207130319c170cd58515 WatchSource:0}: Error finding container f93ed15b10ca9fb9af6ca874e812e9c229ad30fc3be7207130319c170cd58515: Status 404 returned error can't find the container with id f93ed15b10ca9fb9af6ca874e812e9c229ad30fc3be7207130319c170cd58515
Sep 11 17:46:52.551729 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:52.551683    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn" event={"ID":"d650e928-1422-4436-8122-308bcb12e93b","Type":"ContainerStarted","Data":"65c7e41a32beb263899328d8cfebef3ed7494151539add5f71b404f010fbbce4"}
Sep 11 17:46:52.552249 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:52.551755    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn" event={"ID":"d650e928-1422-4436-8122-308bcb12e93b","Type":"ContainerStarted","Data":"f93ed15b10ca9fb9af6ca874e812e9c229ad30fc3be7207130319c170cd58515"}
Sep 11 17:46:52.552249 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:52.551913    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:46:52.553098 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:52.553072    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-5d2qv" event={"ID":"4546e901-4e48-42bd-860c-54b99428070a","Type":"ContainerStarted","Data":"35b8fbf54b72f4a0fd545009df5af19d6d72d5d96f89b5cf39e554e746f4d70b"}
Sep 11 17:46:52.589964 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:52.589878    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn" podStartSLOduration=1.589858092 podStartE2EDuration="1.589858092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:46:52.589850814 +0000 UTC m=+118.034951387" watchObservedRunningTime="2026-09-11 17:46:52.589858092 +0000 UTC m=+118.034958666"
Sep 11 17:46:53.613641 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.613597    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68c4984c9b-mwlqj"]
Sep 11 17:46:53.616713 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.616690    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:46:53.620626 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.620602    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 17:46:53.621457 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.621435    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 17:46:53.621799 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.621761    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 17:46:53.621930 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.621907    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mm5sn\""
Sep 11 17:46:53.623164 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.623140    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 17:46:53.623381 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.623344    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 17:46:53.648600 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.648566    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68c4984c9b-mwlqj"]
Sep 11 17:46:53.650079 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.650045    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 17:46:53.762823 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.762784    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdgcl\" (UniqueName: \"kubernetes.io/projected/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-kube-api-access-mdgcl\") pod \"console-68c4984c9b-mwlqj\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") " pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:46:53.763014 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.762830    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-oauth-serving-cert\") pod \"console-68c4984c9b-mwlqj\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") " pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:46:53.763014 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.762865    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-console-serving-cert\") pod \"console-68c4984c9b-mwlqj\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") " pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:46:53.763014 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.762892    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-console-config\") pod \"console-68c4984c9b-mwlqj\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") " pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:46:53.763014 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.762923    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-service-ca\") pod \"console-68c4984c9b-mwlqj\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") " pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:46:53.763014 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.762966    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-console-oauth-config\") pod \"console-68c4984c9b-mwlqj\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") " pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:46:53.763213 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.763045    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-trusted-ca-bundle\") pod \"console-68c4984c9b-mwlqj\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") " pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:46:53.863788 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.863694    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-service-ca\") pod \"console-68c4984c9b-mwlqj\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") " pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:46:53.863788 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.863754    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-console-oauth-config\") pod \"console-68c4984c9b-mwlqj\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") " pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:46:53.864005 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.863951    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-trusted-ca-bundle\") pod \"console-68c4984c9b-mwlqj\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") " pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:46:53.864371 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.864325    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdgcl\" (UniqueName: \"kubernetes.io/projected/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-kube-api-access-mdgcl\") pod \"console-68c4984c9b-mwlqj\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") " pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:46:53.864511 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.864398    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-oauth-serving-cert\") pod \"console-68c4984c9b-mwlqj\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") " pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:46:53.864511 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.864439    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-console-serving-cert\") pod \"console-68c4984c9b-mwlqj\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") " pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:46:53.864511 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.864469    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-console-config\") pod \"console-68c4984c9b-mwlqj\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") " pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:46:53.864660 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.864568    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-service-ca\") pod \"console-68c4984c9b-mwlqj\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") " pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:46:53.864851 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.864827    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-trusted-ca-bundle\") pod \"console-68c4984c9b-mwlqj\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") " pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:46:53.865096 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.865035    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-oauth-serving-cert\") pod \"console-68c4984c9b-mwlqj\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") " pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:46:53.865173 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.865107    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-console-config\") pod \"console-68c4984c9b-mwlqj\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") " pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:46:53.866755 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.866732    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-console-oauth-config\") pod \"console-68c4984c9b-mwlqj\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") " pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:46:53.867093 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.867078    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-console-serving-cert\") pod \"console-68c4984c9b-mwlqj\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") " pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:46:53.875905 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.875883    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdgcl\" (UniqueName: \"kubernetes.io/projected/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-kube-api-access-mdgcl\") pod \"console-68c4984c9b-mwlqj\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") " pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:46:53.929172 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:53.929127    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:46:54.220038 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.219999    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68c4984c9b-mwlqj"]
Sep 11 17:46:54.223565 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:46:54.223533    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eaf7c5b_64d2_4982_8594_4b09fc8d273e.slice/crio-47d81f72cb6fdf9dbd1a01e362d7bbeb6f3c0e6065e3bef82b5b81b7472908b2 WatchSource:0}: Error finding container 47d81f72cb6fdf9dbd1a01e362d7bbeb6f3c0e6065e3bef82b5b81b7472908b2: Status 404 returned error can't find the container with id 47d81f72cb6fdf9dbd1a01e362d7bbeb6f3c0e6065e3bef82b5b81b7472908b2
Sep 11 17:46:54.562138 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.562054    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c4984c9b-mwlqj" event={"ID":"5eaf7c5b-64d2-4982-8594-4b09fc8d273e","Type":"ContainerStarted","Data":"47d81f72cb6fdf9dbd1a01e362d7bbeb6f3c0e6065e3bef82b5b81b7472908b2"}
Sep 11 17:46:54.698617 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.698574    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-fb9td"]
Sep 11 17:46:54.701529 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.701505    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-fb9td"
Sep 11 17:46:54.704752 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.704725    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 11 17:46:54.706182 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.705572    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 11 17:46:54.706182 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.705804    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 17:46:54.706182 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.706006    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-np6d5\""
Sep 11 17:46:54.706182 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.706039    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 17:46:54.706439 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.706299    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 17:46:54.717967 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.717752    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-fb9td"]
Sep 11 17:46:54.873801 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.873702    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad24ed04-4ced-4532-9c50-aefac7647b00-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-fb9td\" (UID: \"ad24ed04-4ced-4532-9c50-aefac7647b00\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fb9td"
Sep 11 17:46:54.873801 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.873767    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad24ed04-4ced-4532-9c50-aefac7647b00-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-fb9td\" (UID: \"ad24ed04-4ced-4532-9c50-aefac7647b00\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fb9td"
Sep 11 17:46:54.874013 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.873835    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkdhw\" (UniqueName: \"kubernetes.io/projected/ad24ed04-4ced-4532-9c50-aefac7647b00-kube-api-access-hkdhw\") pod \"prometheus-operator-68bcd7f987-fb9td\" (UID: \"ad24ed04-4ced-4532-9c50-aefac7647b00\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fb9td"
Sep 11 17:46:54.874013 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.873865    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24ed04-4ced-4532-9c50-aefac7647b00-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-fb9td\" (UID: \"ad24ed04-4ced-4532-9c50-aefac7647b00\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fb9td"
Sep 11 17:46:54.974291 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.974250    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad24ed04-4ced-4532-9c50-aefac7647b00-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-fb9td\" (UID: \"ad24ed04-4ced-4532-9c50-aefac7647b00\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fb9td"
Sep 11 17:46:54.974498 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.974309    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad24ed04-4ced-4532-9c50-aefac7647b00-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-fb9td\" (UID: \"ad24ed04-4ced-4532-9c50-aefac7647b00\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fb9td"
Sep 11 17:46:54.974498 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.974430    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkdhw\" (UniqueName: \"kubernetes.io/projected/ad24ed04-4ced-4532-9c50-aefac7647b00-kube-api-access-hkdhw\") pod \"prometheus-operator-68bcd7f987-fb9td\" (UID: \"ad24ed04-4ced-4532-9c50-aefac7647b00\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fb9td"
Sep 11 17:46:54.974498 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.974459    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24ed04-4ced-4532-9c50-aefac7647b00-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-fb9td\" (UID: \"ad24ed04-4ced-4532-9c50-aefac7647b00\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fb9td"
Sep 11 17:46:54.978554 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.978307    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 11 17:46:54.978554 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.978322    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 11 17:46:54.979444 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.979424    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 17:46:54.985663 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.985637    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad24ed04-4ced-4532-9c50-aefac7647b00-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-fb9td\" (UID: \"ad24ed04-4ced-4532-9c50-aefac7647b00\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fb9td"
Sep 11 17:46:54.988153 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.988128    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24ed04-4ced-4532-9c50-aefac7647b00-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-fb9td\" (UID: \"ad24ed04-4ced-4532-9c50-aefac7647b00\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fb9td"
Sep 11 17:46:54.988259 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.988156    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad24ed04-4ced-4532-9c50-aefac7647b00-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-fb9td\" (UID: \"ad24ed04-4ced-4532-9c50-aefac7647b00\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fb9td"
Sep 11 17:46:54.988950 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.988929    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 17:46:54.999787 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:54.999762    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 17:46:55.010088 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:55.010015    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkdhw\" (UniqueName: \"kubernetes.io/projected/ad24ed04-4ced-4532-9c50-aefac7647b00-kube-api-access-hkdhw\") pod \"prometheus-operator-68bcd7f987-fb9td\" (UID: \"ad24ed04-4ced-4532-9c50-aefac7647b00\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fb9td"
Sep 11 17:46:55.018205 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:55.018176    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-np6d5\""
Sep 11 17:46:55.024660 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:55.024639    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-fb9td"
Sep 11 17:46:55.321541 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:55.321507    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-fb9td"]
Sep 11 17:46:55.325330 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:46:55.325301    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad24ed04_4ced_4532_9c50_aefac7647b00.slice/crio-4ba068e8270236f4c54f4a69e510672a0a32e6c87d2a34018b5b26731ae9dff6 WatchSource:0}: Error finding container 4ba068e8270236f4c54f4a69e510672a0a32e6c87d2a34018b5b26731ae9dff6: Status 404 returned error can't find the container with id 4ba068e8270236f4c54f4a69e510672a0a32e6c87d2a34018b5b26731ae9dff6
Sep 11 17:46:55.568108 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:55.568060    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-fb9td" event={"ID":"ad24ed04-4ced-4532-9c50-aefac7647b00","Type":"ContainerStarted","Data":"4ba068e8270236f4c54f4a69e510672a0a32e6c87d2a34018b5b26731ae9dff6"}
Sep 11 17:46:58.581438 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:58.581386    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c4984c9b-mwlqj" event={"ID":"5eaf7c5b-64d2-4982-8594-4b09fc8d273e","Type":"ContainerStarted","Data":"61a4bddf25fab267537504de33c562c403388d21f68c5b36da967738d9251b06"}
Sep 11 17:46:58.583578 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:58.583543    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-fb9td" event={"ID":"ad24ed04-4ced-4532-9c50-aefac7647b00","Type":"ContainerStarted","Data":"26bfea58306abcb86650d24d3e59416c4f30dd35f9fca39d3254b74a095ca572"}
Sep 11 17:46:58.583692 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:58.583602    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-fb9td" event={"ID":"ad24ed04-4ced-4532-9c50-aefac7647b00","Type":"ContainerStarted","Data":"a7d13834d08750240855e06ae62f1ddc0410e11ea15d3167a87d2a04c6d3b25e"}
Sep 11 17:46:58.608333 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:58.608281    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68c4984c9b-mwlqj" podStartSLOduration=2.110202927 podStartE2EDuration="5.608265077s" totalImagesPullingTime="3.49806215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:46:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:46:58.607339278 +0000 UTC m=+124.052439850" watchObservedRunningTime="2026-09-11 17:46:58.608265077 +0000 UTC m=+124.053365691"
Sep 11 17:46:58.636676 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:46:58.636611    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-fb9td" podStartSLOduration=2.241420888 podStartE2EDuration="4.636587396s" totalImagesPullingTime="2.395166508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:46:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:46:58.63620149 +0000 UTC m=+124.081302076" watchObservedRunningTime="2026-09-11 17:46:58.636587396 +0000 UTC m=+124.081687968"
Sep 11 17:47:01.519249 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.519210    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6cmmd"]
Sep 11 17:47:01.523369 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.523296    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.527556 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.527532    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 17:47:01.527847 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.527822    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 17:47:01.531858 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.531836    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 17:47:01.532653 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.532629    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hxrkn\""
Sep 11 17:47:01.539712 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.539683    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4ac803ca-e537-4302-848e-ca84419231b4-root\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.539841 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.539740    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ac803ca-e537-4302-848e-ca84419231b4-sys\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.539841 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.539788    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4ac803ca-e537-4302-848e-ca84419231b4-node-exporter-accelerators-collector-config\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.539974 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.539849    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4ac803ca-e537-4302-848e-ca84419231b4-node-exporter-textfile\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.539974 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.539920    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4ac803ca-e537-4302-848e-ca84419231b4-node-exporter-wtmp\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.539974 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.539951    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ac803ca-e537-4302-848e-ca84419231b4-node-exporter-tls\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.540104 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.539983    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzmpj\" (UniqueName: \"kubernetes.io/projected/4ac803ca-e537-4302-848e-ca84419231b4-kube-api-access-rzmpj\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.540104 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.540009    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ac803ca-e537-4302-848e-ca84419231b4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.540104 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.540059    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ac803ca-e537-4302-848e-ca84419231b4-metrics-client-ca\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.641116 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.641071    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ac803ca-e537-4302-848e-ca84419231b4-metrics-client-ca\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.641309 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.641133    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4ac803ca-e537-4302-848e-ca84419231b4-root\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.641309 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.641170    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ac803ca-e537-4302-848e-ca84419231b4-sys\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.641309 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.641209    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4ac803ca-e537-4302-848e-ca84419231b4-node-exporter-accelerators-collector-config\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.641309 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.641246    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4ac803ca-e537-4302-848e-ca84419231b4-node-exporter-textfile\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.641309 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.641253    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4ac803ca-e537-4302-848e-ca84419231b4-root\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.641607 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.641312    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4ac803ca-e537-4302-848e-ca84419231b4-node-exporter-wtmp\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.641607 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.641363    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ac803ca-e537-4302-848e-ca84419231b4-node-exporter-tls\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.641607 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.641404    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzmpj\" (UniqueName: \"kubernetes.io/projected/4ac803ca-e537-4302-848e-ca84419231b4-kube-api-access-rzmpj\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.641607 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.641431    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ac803ca-e537-4302-848e-ca84419231b4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.641607 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.641529    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4ac803ca-e537-4302-848e-ca84419231b4-node-exporter-wtmp\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.641876 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.641777    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ac803ca-e537-4302-848e-ca84419231b4-metrics-client-ca\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.641876 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.641793    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4ac803ca-e537-4302-848e-ca84419231b4-node-exporter-textfile\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.642023 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.641997    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4ac803ca-e537-4302-848e-ca84419231b4-node-exporter-accelerators-collector-config\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.642125 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.642038    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ac803ca-e537-4302-848e-ca84419231b4-sys\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.645012 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.644963    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ac803ca-e537-4302-848e-ca84419231b4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.645199 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.645110    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ac803ca-e537-4302-848e-ca84419231b4-node-exporter-tls\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.668896 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.668859    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzmpj\" (UniqueName: \"kubernetes.io/projected/4ac803ca-e537-4302-848e-ca84419231b4-kube-api-access-rzmpj\") pod \"node-exporter-6cmmd\" (UID: \"4ac803ca-e537-4302-848e-ca84419231b4\") " pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:01.835295 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:01.835215    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6cmmd"
Sep 11 17:47:02.543711 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.543674    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:47:02.547876 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.547850    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.551851 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.551658    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 17:47:02.551851 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.551701    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 17:47:02.551851 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.551721    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 17:47:02.551851 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.551758    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 17:47:02.551851 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.551780    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 17:47:02.551851 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.551706    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 17:47:02.552687 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.552076    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 17:47:02.552687 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.552372    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wv8qx\""
Sep 11 17:47:02.552687 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.552613    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 17:47:02.553098 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.553076    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 17:47:02.571451 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.571420    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:47:02.650889 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.650847    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.651081 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.650905    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.651081 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.650968    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.651081 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.651013    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.651081 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.651051    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jsfh\" (UniqueName: \"kubernetes.io/projected/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-kube-api-access-7jsfh\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.651267 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.651135    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-config-out\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.651267 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.651197    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-config-volume\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.651267 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.651219    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.651267 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.651243    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.651267 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.651261    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.651498 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.651304    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-web-config\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.651498 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.651345    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.651498 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.651389    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.752589 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.752549    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.752786 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.752602    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jsfh\" (UniqueName: \"kubernetes.io/projected/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-kube-api-access-7jsfh\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.752786 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.752644    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-config-out\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.752786 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.752696    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-config-volume\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.752786 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.752721    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.752786 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.752746    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.752786 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.752776    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.753096 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.752815    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-web-config\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.753096 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.752855    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.753096 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.752880    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.753096 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.752933    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.753096 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.752966    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.753096 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.752993    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.754201 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.754171    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.754469 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.754436    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.755601 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.755572    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.755821 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.755793    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.756084 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.756049    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.756175 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.756123    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.756522 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.756410    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-web-config\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.756522 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.756510    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-config-volume\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.756726 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.756680    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-config-out\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.758339 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.758320    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.758737 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.758714    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.758827 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.758742    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.765746 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.765726    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jsfh\" (UniqueName: \"kubernetes.io/projected/d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a-kube-api-access-7jsfh\") pod \"alertmanager-main-0\" (UID: \"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:02.861945 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:02.861855    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:47:03.930158 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:03.930120    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:47:03.930618 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:03.930193    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:47:03.935419 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:03.935388    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:47:04.615076 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:04.615040    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:47:04.772801 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:04.772757    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs\") pod \"network-metrics-daemon-wb9dr\" (UID: \"5ee7b9a9-a331-4b43-b68e-439edfc2dfd6\") " pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:47:04.775311 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:04.775277    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ee7b9a9-a331-4b43-b68e-439edfc2dfd6-metrics-certs\") pod \"network-metrics-daemon-wb9dr\" (UID: \"5ee7b9a9-a331-4b43-b68e-439edfc2dfd6\") " pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:47:05.068316 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:05.068282    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jfwbq\""
Sep 11 17:47:05.075794 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:05.075766    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb9dr"
Sep 11 17:47:06.069381 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.069325    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5759b79cdb-cn79g"]
Sep 11 17:47:06.073133 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.073107    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:06.078839 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.078815    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vcblf\""
Sep 11 17:47:06.079742 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.079486    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 11 17:47:06.079742 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.079515    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 11 17:47:06.079742 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.079541    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 11 17:47:06.079742 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.079594    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 17:47:06.079742 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.079541    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-52f5kot96s1em\""
Sep 11 17:47:06.088310 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.088286    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5759b79cdb-cn79g"]
Sep 11 17:47:06.186920 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.186878    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c21db2eb-ad47-4d40-815e-535bea98791a-secret-metrics-server-tls\") pod \"metrics-server-5759b79cdb-cn79g\" (UID: \"c21db2eb-ad47-4d40-815e-535bea98791a\") " pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:06.187117 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.186940    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c21db2eb-ad47-4d40-815e-535bea98791a-audit-log\") pod \"metrics-server-5759b79cdb-cn79g\" (UID: \"c21db2eb-ad47-4d40-815e-535bea98791a\") " pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:06.187117 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.187031    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c21db2eb-ad47-4d40-815e-535bea98791a-metrics-server-audit-profiles\") pod \"metrics-server-5759b79cdb-cn79g\" (UID: \"c21db2eb-ad47-4d40-815e-535bea98791a\") " pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:06.187117 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.187081    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c21db2eb-ad47-4d40-815e-535bea98791a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5759b79cdb-cn79g\" (UID: \"c21db2eb-ad47-4d40-815e-535bea98791a\") " pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:06.187117 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.187113    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c21db2eb-ad47-4d40-815e-535bea98791a-client-ca-bundle\") pod \"metrics-server-5759b79cdb-cn79g\" (UID: \"c21db2eb-ad47-4d40-815e-535bea98791a\") " pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:06.187336 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.187172    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72dfg\" (UniqueName: \"kubernetes.io/projected/c21db2eb-ad47-4d40-815e-535bea98791a-kube-api-access-72dfg\") pod \"metrics-server-5759b79cdb-cn79g\" (UID: \"c21db2eb-ad47-4d40-815e-535bea98791a\") " pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:06.187336 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.187219    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c21db2eb-ad47-4d40-815e-535bea98791a-secret-metrics-server-client-certs\") pod \"metrics-server-5759b79cdb-cn79g\" (UID: \"c21db2eb-ad47-4d40-815e-535bea98791a\") " pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:06.288448 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.288413    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72dfg\" (UniqueName: \"kubernetes.io/projected/c21db2eb-ad47-4d40-815e-535bea98791a-kube-api-access-72dfg\") pod \"metrics-server-5759b79cdb-cn79g\" (UID: \"c21db2eb-ad47-4d40-815e-535bea98791a\") " pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:06.288652 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.288473    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c21db2eb-ad47-4d40-815e-535bea98791a-secret-metrics-server-client-certs\") pod \"metrics-server-5759b79cdb-cn79g\" (UID: \"c21db2eb-ad47-4d40-815e-535bea98791a\") " pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:06.288652 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.288531    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c21db2eb-ad47-4d40-815e-535bea98791a-secret-metrics-server-tls\") pod \"metrics-server-5759b79cdb-cn79g\" (UID: \"c21db2eb-ad47-4d40-815e-535bea98791a\") " pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:06.288652 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.288572    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c21db2eb-ad47-4d40-815e-535bea98791a-audit-log\") pod \"metrics-server-5759b79cdb-cn79g\" (UID: \"c21db2eb-ad47-4d40-815e-535bea98791a\") " pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:06.288652 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.288616    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c21db2eb-ad47-4d40-815e-535bea98791a-metrics-server-audit-profiles\") pod \"metrics-server-5759b79cdb-cn79g\" (UID: \"c21db2eb-ad47-4d40-815e-535bea98791a\") " pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:06.288652 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.288648    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c21db2eb-ad47-4d40-815e-535bea98791a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5759b79cdb-cn79g\" (UID: \"c21db2eb-ad47-4d40-815e-535bea98791a\") " pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:06.288901 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.288669    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c21db2eb-ad47-4d40-815e-535bea98791a-client-ca-bundle\") pod \"metrics-server-5759b79cdb-cn79g\" (UID: \"c21db2eb-ad47-4d40-815e-535bea98791a\") " pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:06.289088 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.289063    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c21db2eb-ad47-4d40-815e-535bea98791a-audit-log\") pod \"metrics-server-5759b79cdb-cn79g\" (UID: \"c21db2eb-ad47-4d40-815e-535bea98791a\") " pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:06.289783 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.289750    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c21db2eb-ad47-4d40-815e-535bea98791a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5759b79cdb-cn79g\" (UID: \"c21db2eb-ad47-4d40-815e-535bea98791a\") " pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:06.290039 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.290002    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c21db2eb-ad47-4d40-815e-535bea98791a-metrics-server-audit-profiles\") pod \"metrics-server-5759b79cdb-cn79g\" (UID: \"c21db2eb-ad47-4d40-815e-535bea98791a\") " pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:06.291745 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.291721    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c21db2eb-ad47-4d40-815e-535bea98791a-secret-metrics-server-tls\") pod \"metrics-server-5759b79cdb-cn79g\" (UID: \"c21db2eb-ad47-4d40-815e-535bea98791a\") " pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:06.292247 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.292222    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c21db2eb-ad47-4d40-815e-535bea98791a-secret-metrics-server-client-certs\") pod \"metrics-server-5759b79cdb-cn79g\" (UID: \"c21db2eb-ad47-4d40-815e-535bea98791a\") " pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:06.292362 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.292224    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c21db2eb-ad47-4d40-815e-535bea98791a-client-ca-bundle\") pod \"metrics-server-5759b79cdb-cn79g\" (UID: \"c21db2eb-ad47-4d40-815e-535bea98791a\") " pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:06.297930 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.297907    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72dfg\" (UniqueName: \"kubernetes.io/projected/c21db2eb-ad47-4d40-815e-535bea98791a-kube-api-access-72dfg\") pod \"metrics-server-5759b79cdb-cn79g\" (UID: \"c21db2eb-ad47-4d40-815e-535bea98791a\") " pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:06.386513 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:06.386485    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:08.626197 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:47:08.626161    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac803ca_e537_4302_848e_ca84419231b4.slice/crio-f2a1e5a8bada915466d05e181fb898193338a75bfd4baebffffa16dd14fd0cd1 WatchSource:0}: Error finding container f2a1e5a8bada915466d05e181fb898193338a75bfd4baebffffa16dd14fd0cd1: Status 404 returned error can't find the container with id f2a1e5a8bada915466d05e181fb898193338a75bfd4baebffffa16dd14fd0cd1
Sep 11 17:47:08.918338 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:47:08.918299    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee7b9a9_a331_4b43_b68e_439edfc2dfd6.slice/crio-3d3ccae946ecb5157731a91ad9b38c721116264e24fab11db32c71a217017b43 WatchSource:0}: Error finding container 3d3ccae946ecb5157731a91ad9b38c721116264e24fab11db32c71a217017b43: Status 404 returned error can't find the container with id 3d3ccae946ecb5157731a91ad9b38c721116264e24fab11db32c71a217017b43
Sep 11 17:47:08.918885 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:47:08.918726    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc21db2eb_ad47_4d40_815e_535bea98791a.slice/crio-08ac461b35bc9eb43892bd49b7a456180a9fe8340bf9311d5062a3f5966d49d2 WatchSource:0}: Error finding container 08ac461b35bc9eb43892bd49b7a456180a9fe8340bf9311d5062a3f5966d49d2: Status 404 returned error can't find the container with id 08ac461b35bc9eb43892bd49b7a456180a9fe8340bf9311d5062a3f5966d49d2
Sep 11 17:47:08.923135 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:08.923042    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5759b79cdb-cn79g"]
Sep 11 17:47:08.924682 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:08.924658    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wb9dr"]
Sep 11 17:47:08.944515 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:08.944480    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:47:08.948422 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:47:08.948342    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b4a3f0_77ba_4029_bb4b_6e6e66ddbf4a.slice/crio-334b89dba8437eb1eca605b97e4c588e7a64b68a07c7e8d121992996fb5fbc89 WatchSource:0}: Error finding container 334b89dba8437eb1eca605b97e4c588e7a64b68a07c7e8d121992996fb5fbc89: Status 404 returned error can't find the container with id 334b89dba8437eb1eca605b97e4c588e7a64b68a07c7e8d121992996fb5fbc89
Sep 11 17:47:09.633094 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:09.632985    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wb9dr" event={"ID":"5ee7b9a9-a331-4b43-b68e-439edfc2dfd6","Type":"ContainerStarted","Data":"3d3ccae946ecb5157731a91ad9b38c721116264e24fab11db32c71a217017b43"}
Sep 11 17:47:09.635038 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:09.634948    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a","Type":"ContainerStarted","Data":"334b89dba8437eb1eca605b97e4c588e7a64b68a07c7e8d121992996fb5fbc89"}
Sep 11 17:47:09.637860 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:09.637770    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6cmmd" event={"ID":"4ac803ca-e537-4302-848e-ca84419231b4","Type":"ContainerStarted","Data":"f2a1e5a8bada915466d05e181fb898193338a75bfd4baebffffa16dd14fd0cd1"}
Sep 11 17:47:09.640778 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:09.640681    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g" event={"ID":"c21db2eb-ad47-4d40-815e-535bea98791a","Type":"ContainerStarted","Data":"08ac461b35bc9eb43892bd49b7a456180a9fe8340bf9311d5062a3f5966d49d2"}
Sep 11 17:47:09.646392 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:09.646315    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-5d2qv" event={"ID":"4546e901-4e48-42bd-860c-54b99428070a","Type":"ContainerStarted","Data":"5c9169df96285e196975fba30373adec5424f94a513f19f5508d44eefe6195be"}
Sep 11 17:47:09.648894 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:09.648246    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-5d2qv"
Sep 11 17:47:09.660971 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:09.660897    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-5d2qv"
Sep 11 17:47:09.749768 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:09.749091    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-5d2qv" podStartSLOduration=1.7192350809999999 podStartE2EDuration="18.749066502s" totalImagesPullingTime="17.029831421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:46:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:47:09.70708396 +0000 UTC m=+135.152184581" watchObservedRunningTime="2026-09-11 17:47:09.749066502 +0000 UTC m=+135.194167074"
Sep 11 17:47:12.661427 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:12.661382    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g" event={"ID":"c21db2eb-ad47-4d40-815e-535bea98791a","Type":"ContainerStarted","Data":"b208ef03e395e47e966b8c6db63bba6f2a288931b3239512f5a86f65e64a89c5"}
Sep 11 17:47:12.663822 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:12.663786    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wb9dr" event={"ID":"5ee7b9a9-a331-4b43-b68e-439edfc2dfd6","Type":"ContainerStarted","Data":"4bcd681a6652c5bab986b3ae408c6bd1ed9d0d3829eeb92892d764a70d2dce42"}
Sep 11 17:47:12.663948 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:12.663847    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wb9dr" event={"ID":"5ee7b9a9-a331-4b43-b68e-439edfc2dfd6","Type":"ContainerStarted","Data":"a692a483f35c495ea0acc3df6899db82fa9d40b274fcc5c5af6d4c48fcee08f1"}
Sep 11 17:47:12.665969 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:12.665938    2559 generic.go:356] "Generic (PLEG): container finished" podID="d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a" containerID="f9a59a4188211bc602cbf3c9ebce7a73111a3d9324b18c6c54daf34f53cef12c" exitCode=0
Sep 11 17:47:12.666090 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:12.666009    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a","Type":"ContainerDied","Data":"f9a59a4188211bc602cbf3c9ebce7a73111a3d9324b18c6c54daf34f53cef12c"}
Sep 11 17:47:12.667908 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:12.667876    2559 generic.go:356] "Generic (PLEG): container finished" podID="4ac803ca-e537-4302-848e-ca84419231b4" containerID="74eed38254908a6ca9873ddd94afe11bbb96c383f726c0fc0f2edfce8f3a411e" exitCode=0
Sep 11 17:47:12.668037 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:12.667932    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6cmmd" event={"ID":"4ac803ca-e537-4302-848e-ca84419231b4","Type":"ContainerDied","Data":"74eed38254908a6ca9873ddd94afe11bbb96c383f726c0fc0f2edfce8f3a411e"}
Sep 11 17:47:12.693744 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:12.693689    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g" podStartSLOduration=3.639220977 podStartE2EDuration="6.693672799s" totalImagesPullingTime="3.054451822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:47:12.692699404 +0000 UTC m=+138.137799975" watchObservedRunningTime="2026-09-11 17:47:12.693672799 +0000 UTC m=+138.138773371"
Sep 11 17:47:12.750832 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:12.750770    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wb9dr" podStartSLOduration=134.698984171 podStartE2EDuration="2m17.750750906s" totalImagesPullingTime="3.051766735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:47:12.749724924 +0000 UTC m=+138.194825496" watchObservedRunningTime="2026-09-11 17:47:12.750750906 +0000 UTC m=+138.195851479"
Sep 11 17:47:13.562807 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:13.562768    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59cc77dfdd-2vlmn"
Sep 11 17:47:13.678319 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:13.678277    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6cmmd" event={"ID":"4ac803ca-e537-4302-848e-ca84419231b4","Type":"ContainerStarted","Data":"572033b85d1d6d5f4596d0c00355ec258e533df30d184f934dfa107f16a93bfd"}
Sep 11 17:47:13.678841 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:13.678376    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6cmmd" event={"ID":"4ac803ca-e537-4302-848e-ca84419231b4","Type":"ContainerStarted","Data":"b057e3ac20fb89d7ef8d7b56a9d0f3621e2abb7959fb49d2f4575c5b271ba044"}
Sep 11 17:47:13.727245 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:13.727073    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6cmmd" podStartSLOduration=9.292116453 podStartE2EDuration="12.727051458s" totalImagesPullingTime="3.342472877s" totalInitContainerRuntime="92.462128ms" isStatefulPod=true podCreationTimestamp="2026-09-11 17:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:47:13.726376939 +0000 UTC m=+139.171477505" watchObservedRunningTime="2026-09-11 17:47:13.727051458 +0000 UTC m=+139.172152031"
Sep 11 17:47:15.696112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:15.696080    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a","Type":"ContainerStarted","Data":"e3878519228bf4d670aba20194e5bf659dfe99047ec2ffb0900b3d0c04212482"}
Sep 11 17:47:15.696625 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:15.696146    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a","Type":"ContainerStarted","Data":"49584857a6e1cb4eb2f0e6d49e83e25dfb21f6eb14d0d3fd58e8e1ee226b817b"}
Sep 11 17:47:15.696625 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:15.696176    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a","Type":"ContainerStarted","Data":"a71bed0b93dc00cfb4ee368f5817a524b4528f911b0fa7dc1fa5069d81b0fffb"}
Sep 11 17:47:15.696625 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:15.696216    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a","Type":"ContainerStarted","Data":"5bcb56f7323cd37de779895ecebb04669ed11435cc318c40f9a28dada0b01f23"}
Sep 11 17:47:15.696625 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:15.696256    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a","Type":"ContainerStarted","Data":"517ec8203b65ed78c88eab881ef87813562ca1125a2257923edc24351145d4fa"}
Sep 11 17:47:17.709632 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:17.709588    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a","Type":"ContainerStarted","Data":"cdd4e0c665ea8309772845911e9bd1a201ed23ba85a1cfa91ef89c56722ecc42"}
Sep 11 17:47:17.751380 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:17.751292    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=9.14111984 podStartE2EDuration="15.751268292s" totalImagesPullingTime="6.584227382s" totalInitContainerRuntime="25.92107ms" isStatefulPod=false podCreationTimestamp="2026-09-11 17:47:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:47:17.75114495 +0000 UTC m=+143.196245520" watchObservedRunningTime="2026-09-11 17:47:17.751268292 +0000 UTC m=+143.196368864"
Sep 11 17:47:21.266970 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:21.266931    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68c4984c9b-mwlqj"]
Sep 11 17:47:26.386874 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:26.386825    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:26.387280 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:26.386901    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:44.816003 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:44.815967    2559 generic.go:356] "Generic (PLEG): container finished" podID="285adb5b-133b-49bd-a8ff-aaf67c7a3cc6" containerID="6f75b839817fbff74341f2789e4f9cf7f6d84fa15674ef4842795127ef07e7b0" exitCode=0
Sep 11 17:47:44.816541 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:44.816041    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5vb7x" event={"ID":"285adb5b-133b-49bd-a8ff-aaf67c7a3cc6","Type":"ContainerDied","Data":"6f75b839817fbff74341f2789e4f9cf7f6d84fa15674ef4842795127ef07e7b0"}
Sep 11 17:47:44.816541 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:44.816440    2559 scope.go:117] "RemoveContainer" containerID="6f75b839817fbff74341f2789e4f9cf7f6d84fa15674ef4842795127ef07e7b0"
Sep 11 17:47:44.817832 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:44.817811    2559 generic.go:356] "Generic (PLEG): container finished" podID="c539f9b8-d329-47d0-a8ca-79ad211a80c3" containerID="f01e106d3fadcddf03d20894db4f12dbc75f5f5abcd815b20effb0b922d602a6" exitCode=0
Sep 11 17:47:44.817920 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:44.817873    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-lcltf" event={"ID":"c539f9b8-d329-47d0-a8ca-79ad211a80c3","Type":"ContainerDied","Data":"f01e106d3fadcddf03d20894db4f12dbc75f5f5abcd815b20effb0b922d602a6"}
Sep 11 17:47:44.818142 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:44.818131    2559 scope.go:117] "RemoveContainer" containerID="f01e106d3fadcddf03d20894db4f12dbc75f5f5abcd815b20effb0b922d602a6"
Sep 11 17:47:45.824492 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:45.824448    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5vb7x" event={"ID":"285adb5b-133b-49bd-a8ff-aaf67c7a3cc6","Type":"ContainerStarted","Data":"affe69d7cd65f652d0448478d9ac85f1e3abbb01859e1706b0ff5457d9912050"}
Sep 11 17:47:45.826328 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:45.826299    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-lcltf" event={"ID":"c539f9b8-d329-47d0-a8ca-79ad211a80c3","Type":"ContainerStarted","Data":"d8ea65d34f936323a2946a62eb4ecc0ff0d234581f3c37c54fbb5840ca24eb56"}
Sep 11 17:47:46.286890 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.286841    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68c4984c9b-mwlqj" podUID="5eaf7c5b-64d2-4982-8594-4b09fc8d273e" containerName="console" containerID="cri-o://61a4bddf25fab267537504de33c562c403388d21f68c5b36da967738d9251b06" gracePeriod=15
Sep 11 17:47:46.392473 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.392438    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:46.396573 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.396548    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5759b79cdb-cn79g"
Sep 11 17:47:46.562015 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.561946    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:47:46.660461 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.660423    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-console-config\") pod \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") "
Sep 11 17:47:46.660640 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.660474    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-console-serving-cert\") pod \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") "
Sep 11 17:47:46.660640 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.660510    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdgcl\" (UniqueName: \"kubernetes.io/projected/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-kube-api-access-mdgcl\") pod \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") "
Sep 11 17:47:46.660640 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.660536    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-oauth-serving-cert\") pod \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") "
Sep 11 17:47:46.660640 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.660555    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-console-oauth-config\") pod \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") "
Sep 11 17:47:46.660640 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.660636    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-service-ca\") pod \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") "
Sep 11 17:47:46.660871 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.660660    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-trusted-ca-bundle\") pod \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\" (UID: \"5eaf7c5b-64d2-4982-8594-4b09fc8d273e\") "
Sep 11 17:47:46.661003 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.660967    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-console-config" (OuterVolumeSpecName: "console-config") pod "5eaf7c5b-64d2-4982-8594-4b09fc8d273e" (UID: "5eaf7c5b-64d2-4982-8594-4b09fc8d273e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:47:46.661153 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.661045    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5eaf7c5b-64d2-4982-8594-4b09fc8d273e" (UID: "5eaf7c5b-64d2-4982-8594-4b09fc8d273e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:47:46.661153 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.661138    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5eaf7c5b-64d2-4982-8594-4b09fc8d273e" (UID: "5eaf7c5b-64d2-4982-8594-4b09fc8d273e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:47:46.661264 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.661176    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-service-ca" (OuterVolumeSpecName: "service-ca") pod "5eaf7c5b-64d2-4982-8594-4b09fc8d273e" (UID: "5eaf7c5b-64d2-4982-8594-4b09fc8d273e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:47:46.662941 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.662912    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5eaf7c5b-64d2-4982-8594-4b09fc8d273e" (UID: "5eaf7c5b-64d2-4982-8594-4b09fc8d273e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:47:46.663043 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.662970    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-kube-api-access-mdgcl" (OuterVolumeSpecName: "kube-api-access-mdgcl") pod "5eaf7c5b-64d2-4982-8594-4b09fc8d273e" (UID: "5eaf7c5b-64d2-4982-8594-4b09fc8d273e"). InnerVolumeSpecName "kube-api-access-mdgcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:47:46.663100 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.663058    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5eaf7c5b-64d2-4982-8594-4b09fc8d273e" (UID: "5eaf7c5b-64d2-4982-8594-4b09fc8d273e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:47:46.761718 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.761677    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-service-ca\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:47:46.761718 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.761716    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-trusted-ca-bundle\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:47:46.762128 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.761729    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-console-config\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:47:46.762128 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.761742    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-console-serving-cert\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:47:46.762128 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.761753    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdgcl\" (UniqueName: \"kubernetes.io/projected/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-kube-api-access-mdgcl\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:47:46.762128 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.761766    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-oauth-serving-cert\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:47:46.762128 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.761777    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5eaf7c5b-64d2-4982-8594-4b09fc8d273e-console-oauth-config\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:47:46.831302 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.831223    2559 generic.go:356] "Generic (PLEG): container finished" podID="5eaf7c5b-64d2-4982-8594-4b09fc8d273e" containerID="61a4bddf25fab267537504de33c562c403388d21f68c5b36da967738d9251b06" exitCode=0
Sep 11 17:47:46.831302 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.831269    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c4984c9b-mwlqj"
Sep 11 17:47:46.831757 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.831315    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c4984c9b-mwlqj" event={"ID":"5eaf7c5b-64d2-4982-8594-4b09fc8d273e","Type":"ContainerDied","Data":"61a4bddf25fab267537504de33c562c403388d21f68c5b36da967738d9251b06"}
Sep 11 17:47:46.831757 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.831384    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c4984c9b-mwlqj" event={"ID":"5eaf7c5b-64d2-4982-8594-4b09fc8d273e","Type":"ContainerDied","Data":"47d81f72cb6fdf9dbd1a01e362d7bbeb6f3c0e6065e3bef82b5b81b7472908b2"}
Sep 11 17:47:46.831757 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.831410    2559 scope.go:117] "RemoveContainer" containerID="61a4bddf25fab267537504de33c562c403388d21f68c5b36da967738d9251b06"
Sep 11 17:47:46.840824 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.840804    2559 scope.go:117] "RemoveContainer" containerID="61a4bddf25fab267537504de33c562c403388d21f68c5b36da967738d9251b06"
Sep 11 17:47:46.841139 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:47:46.841118    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61a4bddf25fab267537504de33c562c403388d21f68c5b36da967738d9251b06\": container with ID starting with 61a4bddf25fab267537504de33c562c403388d21f68c5b36da967738d9251b06 not found: ID does not exist" containerID="61a4bddf25fab267537504de33c562c403388d21f68c5b36da967738d9251b06"
Sep 11 17:47:46.841200 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.841147    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61a4bddf25fab267537504de33c562c403388d21f68c5b36da967738d9251b06"} err="failed to get container status \"61a4bddf25fab267537504de33c562c403388d21f68c5b36da967738d9251b06\": rpc error: code = NotFound desc = could not find container \"61a4bddf25fab267537504de33c562c403388d21f68c5b36da967738d9251b06\": container with ID starting with 61a4bddf25fab267537504de33c562c403388d21f68c5b36da967738d9251b06 not found: ID does not exist"
Sep 11 17:47:46.853681 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.853654    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68c4984c9b-mwlqj"]
Sep 11 17:47:46.857398 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:46.857374    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68c4984c9b-mwlqj"]
Sep 11 17:47:47.052328 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:47.052284    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5eaf7c5b-64d2-4982-8594-4b09fc8d273e" path="/var/lib/kubelet/pods/5eaf7c5b-64d2-4982-8594-4b09fc8d273e/volumes"
Sep 11 17:47:55.865737 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:55.865700    2559 generic.go:356] "Generic (PLEG): container finished" podID="abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2" containerID="d198dc14c829b8c17a8c6d0d607c7c968a95420f98ebd3473a67408096a32bc8" exitCode=0
Sep 11 17:47:55.866140 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:55.865780    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4hw4h" event={"ID":"abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2","Type":"ContainerDied","Data":"d198dc14c829b8c17a8c6d0d607c7c968a95420f98ebd3473a67408096a32bc8"}
Sep 11 17:47:55.866185 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:55.866160    2559 scope.go:117] "RemoveContainer" containerID="d198dc14c829b8c17a8c6d0d607c7c968a95420f98ebd3473a67408096a32bc8"
Sep 11 17:47:56.871924 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:47:56.871894    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4hw4h" event={"ID":"abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2","Type":"ContainerStarted","Data":"dbc1d38aeb1900bc7a80d021666a02f2122affa3aa428c2df83d99776df5073a"}
Sep 11 17:48:25.932344 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:25.932301    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5865d99ff4-rswtx"]
Sep 11 17:48:25.932975 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:25.932746    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eaf7c5b-64d2-4982-8594-4b09fc8d273e" containerName="console"
Sep 11 17:48:25.932975 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:25.932764    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eaf7c5b-64d2-4982-8594-4b09fc8d273e" containerName="console"
Sep 11 17:48:25.932975 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:25.932849    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eaf7c5b-64d2-4982-8594-4b09fc8d273e" containerName="console"
Sep 11 17:48:25.935181 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:25.935157    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:25.939589 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:25.939559    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 11 17:48:25.939732 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:25.939633    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 11 17:48:25.939732 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:25.939637    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 11 17:48:25.939732 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:25.939637    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 11 17:48:25.940254 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:25.940225    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-qjjtl\""
Sep 11 17:48:25.940383 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:25.940365    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 11 17:48:25.948497 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:25.948474    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 11 17:48:25.964795 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:25.964762    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5865d99ff4-rswtx"]
Sep 11 17:48:26.013182 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.013146    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b834f53-64b2-45b3-95b9-e148f9aef7cf-serving-certs-ca-bundle\") pod \"telemeter-client-5865d99ff4-rswtx\" (UID: \"2b834f53-64b2-45b3-95b9-e148f9aef7cf\") " pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.013391 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.013194    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2b834f53-64b2-45b3-95b9-e148f9aef7cf-telemeter-client-tls\") pod \"telemeter-client-5865d99ff4-rswtx\" (UID: \"2b834f53-64b2-45b3-95b9-e148f9aef7cf\") " pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.013391 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.013237    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b834f53-64b2-45b3-95b9-e148f9aef7cf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5865d99ff4-rswtx\" (UID: \"2b834f53-64b2-45b3-95b9-e148f9aef7cf\") " pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.013391 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.013257    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b834f53-64b2-45b3-95b9-e148f9aef7cf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5865d99ff4-rswtx\" (UID: \"2b834f53-64b2-45b3-95b9-e148f9aef7cf\") " pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.013391 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.013276    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2b834f53-64b2-45b3-95b9-e148f9aef7cf-federate-client-tls\") pod \"telemeter-client-5865d99ff4-rswtx\" (UID: \"2b834f53-64b2-45b3-95b9-e148f9aef7cf\") " pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.013391 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.013319    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2b834f53-64b2-45b3-95b9-e148f9aef7cf-secret-telemeter-client\") pod \"telemeter-client-5865d99ff4-rswtx\" (UID: \"2b834f53-64b2-45b3-95b9-e148f9aef7cf\") " pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.013391 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.013342    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b834f53-64b2-45b3-95b9-e148f9aef7cf-metrics-client-ca\") pod \"telemeter-client-5865d99ff4-rswtx\" (UID: \"2b834f53-64b2-45b3-95b9-e148f9aef7cf\") " pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.013391 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.013384    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jrj8\" (UniqueName: \"kubernetes.io/projected/2b834f53-64b2-45b3-95b9-e148f9aef7cf-kube-api-access-9jrj8\") pod \"telemeter-client-5865d99ff4-rswtx\" (UID: \"2b834f53-64b2-45b3-95b9-e148f9aef7cf\") " pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.114378 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.114303    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2b834f53-64b2-45b3-95b9-e148f9aef7cf-secret-telemeter-client\") pod \"telemeter-client-5865d99ff4-rswtx\" (UID: \"2b834f53-64b2-45b3-95b9-e148f9aef7cf\") " pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.114567 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.114412    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b834f53-64b2-45b3-95b9-e148f9aef7cf-metrics-client-ca\") pod \"telemeter-client-5865d99ff4-rswtx\" (UID: \"2b834f53-64b2-45b3-95b9-e148f9aef7cf\") " pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.114567 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.114439    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jrj8\" (UniqueName: \"kubernetes.io/projected/2b834f53-64b2-45b3-95b9-e148f9aef7cf-kube-api-access-9jrj8\") pod \"telemeter-client-5865d99ff4-rswtx\" (UID: \"2b834f53-64b2-45b3-95b9-e148f9aef7cf\") " pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.114567 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.114548    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b834f53-64b2-45b3-95b9-e148f9aef7cf-serving-certs-ca-bundle\") pod \"telemeter-client-5865d99ff4-rswtx\" (UID: \"2b834f53-64b2-45b3-95b9-e148f9aef7cf\") " pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.114697 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.114623    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2b834f53-64b2-45b3-95b9-e148f9aef7cf-telemeter-client-tls\") pod \"telemeter-client-5865d99ff4-rswtx\" (UID: \"2b834f53-64b2-45b3-95b9-e148f9aef7cf\") " pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.114697 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.114682    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b834f53-64b2-45b3-95b9-e148f9aef7cf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5865d99ff4-rswtx\" (UID: \"2b834f53-64b2-45b3-95b9-e148f9aef7cf\") " pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.114794 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.114721    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b834f53-64b2-45b3-95b9-e148f9aef7cf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5865d99ff4-rswtx\" (UID: \"2b834f53-64b2-45b3-95b9-e148f9aef7cf\") " pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.114794 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.114757    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2b834f53-64b2-45b3-95b9-e148f9aef7cf-federate-client-tls\") pod \"telemeter-client-5865d99ff4-rswtx\" (UID: \"2b834f53-64b2-45b3-95b9-e148f9aef7cf\") " pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.115376 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.115333    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b834f53-64b2-45b3-95b9-e148f9aef7cf-serving-certs-ca-bundle\") pod \"telemeter-client-5865d99ff4-rswtx\" (UID: \"2b834f53-64b2-45b3-95b9-e148f9aef7cf\") " pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.115559 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.115534    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b834f53-64b2-45b3-95b9-e148f9aef7cf-metrics-client-ca\") pod \"telemeter-client-5865d99ff4-rswtx\" (UID: \"2b834f53-64b2-45b3-95b9-e148f9aef7cf\") " pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.115657 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.115639    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b834f53-64b2-45b3-95b9-e148f9aef7cf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5865d99ff4-rswtx\" (UID: \"2b834f53-64b2-45b3-95b9-e148f9aef7cf\") " pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.117187 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.117158    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2b834f53-64b2-45b3-95b9-e148f9aef7cf-secret-telemeter-client\") pod \"telemeter-client-5865d99ff4-rswtx\" (UID: \"2b834f53-64b2-45b3-95b9-e148f9aef7cf\") " pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.117303 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.117218    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2b834f53-64b2-45b3-95b9-e148f9aef7cf-federate-client-tls\") pod \"telemeter-client-5865d99ff4-rswtx\" (UID: \"2b834f53-64b2-45b3-95b9-e148f9aef7cf\") " pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.117478 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.117457    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b834f53-64b2-45b3-95b9-e148f9aef7cf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5865d99ff4-rswtx\" (UID: \"2b834f53-64b2-45b3-95b9-e148f9aef7cf\") " pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.117552 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.117529    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2b834f53-64b2-45b3-95b9-e148f9aef7cf-telemeter-client-tls\") pod \"telemeter-client-5865d99ff4-rswtx\" (UID: \"2b834f53-64b2-45b3-95b9-e148f9aef7cf\") " pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.127325 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.127301    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jrj8\" (UniqueName: \"kubernetes.io/projected/2b834f53-64b2-45b3-95b9-e148f9aef7cf-kube-api-access-9jrj8\") pod \"telemeter-client-5865d99ff4-rswtx\" (UID: \"2b834f53-64b2-45b3-95b9-e148f9aef7cf\") " pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.248254 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.248157    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx"
Sep 11 17:48:26.545080 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.545009    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5865d99ff4-rswtx"]
Sep 11 17:48:26.548462 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:48:26.548434    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b834f53_64b2_45b3_95b9_e148f9aef7cf.slice/crio-0a4246e7ed037fcb502e8eb0137a55b075f0140b09adf69eb1ffd928497eff1e WatchSource:0}: Error finding container 0a4246e7ed037fcb502e8eb0137a55b075f0140b09adf69eb1ffd928497eff1e: Status 404 returned error can't find the container with id 0a4246e7ed037fcb502e8eb0137a55b075f0140b09adf69eb1ffd928497eff1e
Sep 11 17:48:26.986020 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:26.985986    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx" event={"ID":"2b834f53-64b2-45b3-95b9-e148f9aef7cf","Type":"ContainerStarted","Data":"0a4246e7ed037fcb502e8eb0137a55b075f0140b09adf69eb1ffd928497eff1e"}
Sep 11 17:48:28.996861 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:28.996821    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx" event={"ID":"2b834f53-64b2-45b3-95b9-e148f9aef7cf","Type":"ContainerStarted","Data":"3b32c0b5802af1a9545e08918b267960a44eecdaa2c5a0c7a0d9d12072aedf69"}
Sep 11 17:48:28.997379 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:28.996907    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx" event={"ID":"2b834f53-64b2-45b3-95b9-e148f9aef7cf","Type":"ContainerStarted","Data":"26c2094870429a76f00ddfff47d47d720fa448b69d480e2da5f3b3f1d0f29fa0"}
Sep 11 17:48:28.997379 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:28.996926    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx" event={"ID":"2b834f53-64b2-45b3-95b9-e148f9aef7cf","Type":"ContainerStarted","Data":"30a0716ec7f48cb4d8f3a06050c71fdfd9c26138aa873cb956d20eb5e8ea3151"}
Sep 11 17:48:29.028084 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.028028    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5865d99ff4-rswtx" podStartSLOduration=2.252510547 podStartE2EDuration="4.02801431s" totalImagesPullingTime="1.775503763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:48:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:48:29.027762676 +0000 UTC m=+214.472863248" watchObservedRunningTime="2026-09-11 17:48:29.02801431 +0000 UTC m=+214.473114881"
Sep 11 17:48:29.139148 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.139114    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59979fd599-gthwh"]
Sep 11 17:48:29.141199 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.141181    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:48:29.143884 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.143860    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 17:48:29.144000 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.143901    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 17:48:29.144286 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.144268    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mm5sn\""
Sep 11 17:48:29.144602 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.144576    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 17:48:29.144699 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.144678    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 17:48:29.146389 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.146369    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 17:48:29.151553 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.151531    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 17:48:29.161848 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.161819    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59979fd599-gthwh"]
Sep 11 17:48:29.242746 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.242704    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ee048b1-fc58-44cf-a1ec-2e45209ea089-console-serving-cert\") pod \"console-59979fd599-gthwh\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") " pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:48:29.242746 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.242745    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ee048b1-fc58-44cf-a1ec-2e45209ea089-console-config\") pod \"console-59979fd599-gthwh\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") " pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:48:29.242970 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.242770    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ee048b1-fc58-44cf-a1ec-2e45209ea089-trusted-ca-bundle\") pod \"console-59979fd599-gthwh\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") " pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:48:29.242970 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.242792    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjfcq\" (UniqueName: \"kubernetes.io/projected/5ee048b1-fc58-44cf-a1ec-2e45209ea089-kube-api-access-tjfcq\") pod \"console-59979fd599-gthwh\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") " pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:48:29.242970 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.242891    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ee048b1-fc58-44cf-a1ec-2e45209ea089-oauth-serving-cert\") pod \"console-59979fd599-gthwh\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") " pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:48:29.242970 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.242925    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ee048b1-fc58-44cf-a1ec-2e45209ea089-console-oauth-config\") pod \"console-59979fd599-gthwh\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") " pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:48:29.242970 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.242960    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ee048b1-fc58-44cf-a1ec-2e45209ea089-service-ca\") pod \"console-59979fd599-gthwh\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") " pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:48:29.344235 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.344133    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ee048b1-fc58-44cf-a1ec-2e45209ea089-oauth-serving-cert\") pod \"console-59979fd599-gthwh\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") " pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:48:29.344235 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.344178    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ee048b1-fc58-44cf-a1ec-2e45209ea089-console-oauth-config\") pod \"console-59979fd599-gthwh\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") " pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:48:29.344235 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.344209    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ee048b1-fc58-44cf-a1ec-2e45209ea089-service-ca\") pod \"console-59979fd599-gthwh\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") " pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:48:29.344553 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.344258    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ee048b1-fc58-44cf-a1ec-2e45209ea089-console-serving-cert\") pod \"console-59979fd599-gthwh\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") " pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:48:29.344553 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.344274    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ee048b1-fc58-44cf-a1ec-2e45209ea089-console-config\") pod \"console-59979fd599-gthwh\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") " pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:48:29.344553 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.344293    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ee048b1-fc58-44cf-a1ec-2e45209ea089-trusted-ca-bundle\") pod \"console-59979fd599-gthwh\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") " pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:48:29.344553 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.344316    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjfcq\" (UniqueName: \"kubernetes.io/projected/5ee048b1-fc58-44cf-a1ec-2e45209ea089-kube-api-access-tjfcq\") pod \"console-59979fd599-gthwh\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") " pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:48:29.345016 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.344990    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ee048b1-fc58-44cf-a1ec-2e45209ea089-oauth-serving-cert\") pod \"console-59979fd599-gthwh\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") " pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:48:29.345125 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.344994    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ee048b1-fc58-44cf-a1ec-2e45209ea089-service-ca\") pod \"console-59979fd599-gthwh\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") " pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:48:29.345208 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.345183    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ee048b1-fc58-44cf-a1ec-2e45209ea089-console-config\") pod \"console-59979fd599-gthwh\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") " pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:48:29.345249 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.345217    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ee048b1-fc58-44cf-a1ec-2e45209ea089-trusted-ca-bundle\") pod \"console-59979fd599-gthwh\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") " pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:48:29.346783 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.346765    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ee048b1-fc58-44cf-a1ec-2e45209ea089-console-serving-cert\") pod \"console-59979fd599-gthwh\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") " pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:48:29.346904 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.346885    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ee048b1-fc58-44cf-a1ec-2e45209ea089-console-oauth-config\") pod \"console-59979fd599-gthwh\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") " pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:48:29.353418 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.353398    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjfcq\" (UniqueName: \"kubernetes.io/projected/5ee048b1-fc58-44cf-a1ec-2e45209ea089-kube-api-access-tjfcq\") pod \"console-59979fd599-gthwh\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") " pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:48:29.450734 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.450689    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:48:29.736332 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:29.736305    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59979fd599-gthwh"]
Sep 11 17:48:29.738465 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:48:29.738433    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee048b1_fc58_44cf_a1ec_2e45209ea089.slice/crio-1daf019ac1bd2d6cb896339863fbbeb6b24a1643a720670d29974f6e29ca8ae6 WatchSource:0}: Error finding container 1daf019ac1bd2d6cb896339863fbbeb6b24a1643a720670d29974f6e29ca8ae6: Status 404 returned error can't find the container with id 1daf019ac1bd2d6cb896339863fbbeb6b24a1643a720670d29974f6e29ca8ae6
Sep 11 17:48:30.004712 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:30.004605    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59979fd599-gthwh" event={"ID":"5ee048b1-fc58-44cf-a1ec-2e45209ea089","Type":"ContainerStarted","Data":"c6c519effd8b42bcb9a7cd7a77f10bb208c9feee1270e6aa4c981f852b6d8206"}
Sep 11 17:48:30.004712 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:30.004660    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59979fd599-gthwh" event={"ID":"5ee048b1-fc58-44cf-a1ec-2e45209ea089","Type":"ContainerStarted","Data":"1daf019ac1bd2d6cb896339863fbbeb6b24a1643a720670d29974f6e29ca8ae6"}
Sep 11 17:48:30.036619 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:30.036556    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59979fd599-gthwh" podStartSLOduration=1.036537846 podStartE2EDuration="1.036537846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:48:30.035389568 +0000 UTC m=+215.480490140" watchObservedRunningTime="2026-09-11 17:48:30.036537846 +0000 UTC m=+215.481638454"
Sep 11 17:48:37.037410 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.037379    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59979fd599-gthwh"]
Sep 11 17:48:37.076996 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.076965    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d7c75f5d6-7xbxm"]
Sep 11 17:48:37.079777 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.079760    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:37.093978 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.093946    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d7c75f5d6-7xbxm"]
Sep 11 17:48:37.111498 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.111466    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-console-oauth-config\") pod \"console-d7c75f5d6-7xbxm\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") " pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:37.111498 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.111498    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-trusted-ca-bundle\") pod \"console-d7c75f5d6-7xbxm\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") " pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:37.111727 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.111615    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-console-config\") pod \"console-d7c75f5d6-7xbxm\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") " pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:37.111803 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.111787    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnl46\" (UniqueName: \"kubernetes.io/projected/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-kube-api-access-vnl46\") pod \"console-d7c75f5d6-7xbxm\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") " pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:37.111856 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.111812    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-oauth-serving-cert\") pod \"console-d7c75f5d6-7xbxm\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") " pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:37.112056 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.112023    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-console-serving-cert\") pod \"console-d7c75f5d6-7xbxm\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") " pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:37.112200 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.112067    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-service-ca\") pod \"console-d7c75f5d6-7xbxm\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") " pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:37.212541 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.212500    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnl46\" (UniqueName: \"kubernetes.io/projected/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-kube-api-access-vnl46\") pod \"console-d7c75f5d6-7xbxm\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") " pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:37.212541 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.212542    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-oauth-serving-cert\") pod \"console-d7c75f5d6-7xbxm\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") " pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:37.212756 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.212582    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-console-serving-cert\") pod \"console-d7c75f5d6-7xbxm\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") " pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:37.212756 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.212606    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-service-ca\") pod \"console-d7c75f5d6-7xbxm\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") " pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:37.212844 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.212751    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-console-oauth-config\") pod \"console-d7c75f5d6-7xbxm\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") " pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:37.212844 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.212785    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-trusted-ca-bundle\") pod \"console-d7c75f5d6-7xbxm\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") " pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:37.212844 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.212831    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-console-config\") pod \"console-d7c75f5d6-7xbxm\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") " pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:37.213481 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.213452    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-oauth-serving-cert\") pod \"console-d7c75f5d6-7xbxm\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") " pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:37.213604 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.213520    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-service-ca\") pod \"console-d7c75f5d6-7xbxm\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") " pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:37.213650 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.213602    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-console-config\") pod \"console-d7c75f5d6-7xbxm\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") " pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:37.213764 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.213746    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-trusted-ca-bundle\") pod \"console-d7c75f5d6-7xbxm\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") " pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:37.215203 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.215177    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-console-serving-cert\") pod \"console-d7c75f5d6-7xbxm\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") " pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:37.215203 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.215194    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-console-oauth-config\") pod \"console-d7c75f5d6-7xbxm\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") " pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:37.220752 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.220728    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnl46\" (UniqueName: \"kubernetes.io/projected/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-kube-api-access-vnl46\") pod \"console-d7c75f5d6-7xbxm\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") " pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:37.389093 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.389062    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:37.661138 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:37.661113    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d7c75f5d6-7xbxm"]
Sep 11 17:48:37.663512 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:48:37.663477    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8ce2d4_dbbb_4189_ba9d_94c6363efcdd.slice/crio-f541c825e4cd1ed21e15ec1e6bb8b51b00e2ae3954e68c9370c78bc3f3dd3ca4 WatchSource:0}: Error finding container f541c825e4cd1ed21e15ec1e6bb8b51b00e2ae3954e68c9370c78bc3f3dd3ca4: Status 404 returned error can't find the container with id f541c825e4cd1ed21e15ec1e6bb8b51b00e2ae3954e68c9370c78bc3f3dd3ca4
Sep 11 17:48:38.049572 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:38.049483    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d7c75f5d6-7xbxm" event={"ID":"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd","Type":"ContainerStarted","Data":"fe7e3b7af966a8021e9339be1e90d069c5061e093b9bf9c2f951f29579bf5ddf"}
Sep 11 17:48:38.049572 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:38.049525    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d7c75f5d6-7xbxm" event={"ID":"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd","Type":"ContainerStarted","Data":"f541c825e4cd1ed21e15ec1e6bb8b51b00e2ae3954e68c9370c78bc3f3dd3ca4"}
Sep 11 17:48:38.069241 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:38.069190    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d7c75f5d6-7xbxm" podStartSLOduration=1.069173579 podStartE2EDuration="1.069173579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:48:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:48:38.06770595 +0000 UTC m=+223.512806522" watchObservedRunningTime="2026-09-11 17:48:38.069173579 +0000 UTC m=+223.514274149"
Sep 11 17:48:39.451843 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:39.451807    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:48:47.389618 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:47.389577    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:47.390014 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:47.389632    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:47.394666 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:47.394644    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:48:48.091726 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:48:48.091696    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:49:02.056115 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:02.056054    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59979fd599-gthwh" podUID="5ee048b1-fc58-44cf-a1ec-2e45209ea089" containerName="console" containerID="cri-o://c6c519effd8b42bcb9a7cd7a77f10bb208c9feee1270e6aa4c981f852b6d8206" gracePeriod=15
Sep 11 17:49:02.300988 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:02.300964    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:49:02.430556 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:02.430518    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ee048b1-fc58-44cf-a1ec-2e45209ea089-console-oauth-config\") pod \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") "
Sep 11 17:49:02.430556 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:02.430561    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ee048b1-fc58-44cf-a1ec-2e45209ea089-console-serving-cert\") pod \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") "
Sep 11 17:49:02.430812 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:02.430589    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ee048b1-fc58-44cf-a1ec-2e45209ea089-oauth-serving-cert\") pod \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") "
Sep 11 17:49:02.430812 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:02.430617    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ee048b1-fc58-44cf-a1ec-2e45209ea089-console-config\") pod \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") "
Sep 11 17:49:02.430812 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:02.430639    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjfcq\" (UniqueName: \"kubernetes.io/projected/5ee048b1-fc58-44cf-a1ec-2e45209ea089-kube-api-access-tjfcq\") pod \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") "
Sep 11 17:49:02.430812 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:02.430664    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ee048b1-fc58-44cf-a1ec-2e45209ea089-trusted-ca-bundle\") pod \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") "
Sep 11 17:49:02.430812 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:02.430693    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ee048b1-fc58-44cf-a1ec-2e45209ea089-service-ca\") pod \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\" (UID: \"5ee048b1-fc58-44cf-a1ec-2e45209ea089\") "
Sep 11 17:49:02.431138 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:02.431092    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee048b1-fc58-44cf-a1ec-2e45209ea089-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5ee048b1-fc58-44cf-a1ec-2e45209ea089" (UID: "5ee048b1-fc58-44cf-a1ec-2e45209ea089"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:49:02.431218 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:02.431125    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee048b1-fc58-44cf-a1ec-2e45209ea089-console-config" (OuterVolumeSpecName: "console-config") pod "5ee048b1-fc58-44cf-a1ec-2e45209ea089" (UID: "5ee048b1-fc58-44cf-a1ec-2e45209ea089"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:49:02.431218 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:02.431130    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee048b1-fc58-44cf-a1ec-2e45209ea089-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5ee048b1-fc58-44cf-a1ec-2e45209ea089" (UID: "5ee048b1-fc58-44cf-a1ec-2e45209ea089"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:49:02.431477 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:02.431291    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ee048b1-fc58-44cf-a1ec-2e45209ea089-console-config\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:49:02.431477 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:02.431339    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ee048b1-fc58-44cf-a1ec-2e45209ea089-trusted-ca-bundle\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:49:02.431477 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:02.431371    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ee048b1-fc58-44cf-a1ec-2e45209ea089-oauth-serving-cert\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:49:02.431634 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:02.431558    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee048b1-fc58-44cf-a1ec-2e45209ea089-service-ca" (OuterVolumeSpecName: "service-ca") pod "5ee048b1-fc58-44cf-a1ec-2e45209ea089" (UID: "5ee048b1-fc58-44cf-a1ec-2e45209ea089"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:49:02.432935 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:02.432906    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee048b1-fc58-44cf-a1ec-2e45209ea089-kube-api-access-tjfcq" (OuterVolumeSpecName: "kube-api-access-tjfcq") pod "5ee048b1-fc58-44cf-a1ec-2e45209ea089" (UID: "5ee048b1-fc58-44cf-a1ec-2e45209ea089"). InnerVolumeSpecName "kube-api-access-tjfcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:49:02.432935 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:02.432925    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ee048b1-fc58-44cf-a1ec-2e45209ea089-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5ee048b1-fc58-44cf-a1ec-2e45209ea089" (UID: "5ee048b1-fc58-44cf-a1ec-2e45209ea089"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:49:02.433066 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:02.432912    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ee048b1-fc58-44cf-a1ec-2e45209ea089-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5ee048b1-fc58-44cf-a1ec-2e45209ea089" (UID: "5ee048b1-fc58-44cf-a1ec-2e45209ea089"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:49:02.532627 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:02.532585    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ee048b1-fc58-44cf-a1ec-2e45209ea089-service-ca\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:49:02.532627 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:02.532619    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ee048b1-fc58-44cf-a1ec-2e45209ea089-console-oauth-config\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:49:02.532627 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:02.532629    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ee048b1-fc58-44cf-a1ec-2e45209ea089-console-serving-cert\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:49:02.532862 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:02.532638    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjfcq\" (UniqueName: \"kubernetes.io/projected/5ee048b1-fc58-44cf-a1ec-2e45209ea089-kube-api-access-tjfcq\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:49:03.149717 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:03.149678    2559 generic.go:356] "Generic (PLEG): container finished" podID="5ee048b1-fc58-44cf-a1ec-2e45209ea089" containerID="c6c519effd8b42bcb9a7cd7a77f10bb208c9feee1270e6aa4c981f852b6d8206" exitCode=0
Sep 11 17:49:03.150166 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:03.149728    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59979fd599-gthwh"
Sep 11 17:49:03.150166 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:03.149754    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59979fd599-gthwh" event={"ID":"5ee048b1-fc58-44cf-a1ec-2e45209ea089","Type":"ContainerDied","Data":"c6c519effd8b42bcb9a7cd7a77f10bb208c9feee1270e6aa4c981f852b6d8206"}
Sep 11 17:49:03.150166 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:03.149800    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59979fd599-gthwh" event={"ID":"5ee048b1-fc58-44cf-a1ec-2e45209ea089","Type":"ContainerDied","Data":"1daf019ac1bd2d6cb896339863fbbeb6b24a1643a720670d29974f6e29ca8ae6"}
Sep 11 17:49:03.150166 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:03.149826    2559 scope.go:117] "RemoveContainer" containerID="c6c519effd8b42bcb9a7cd7a77f10bb208c9feee1270e6aa4c981f852b6d8206"
Sep 11 17:49:03.158891 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:03.158867    2559 scope.go:117] "RemoveContainer" containerID="c6c519effd8b42bcb9a7cd7a77f10bb208c9feee1270e6aa4c981f852b6d8206"
Sep 11 17:49:03.159186 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:49:03.159165    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6c519effd8b42bcb9a7cd7a77f10bb208c9feee1270e6aa4c981f852b6d8206\": container with ID starting with c6c519effd8b42bcb9a7cd7a77f10bb208c9feee1270e6aa4c981f852b6d8206 not found: ID does not exist" containerID="c6c519effd8b42bcb9a7cd7a77f10bb208c9feee1270e6aa4c981f852b6d8206"
Sep 11 17:49:03.159239 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:03.159194    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6c519effd8b42bcb9a7cd7a77f10bb208c9feee1270e6aa4c981f852b6d8206"} err="failed to get container status \"c6c519effd8b42bcb9a7cd7a77f10bb208c9feee1270e6aa4c981f852b6d8206\": rpc error: code = NotFound desc = could not find container \"c6c519effd8b42bcb9a7cd7a77f10bb208c9feee1270e6aa4c981f852b6d8206\": container with ID starting with c6c519effd8b42bcb9a7cd7a77f10bb208c9feee1270e6aa4c981f852b6d8206 not found: ID does not exist"
Sep 11 17:49:03.165471 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:03.165444    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59979fd599-gthwh"]
Sep 11 17:49:03.168737 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:03.168716    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59979fd599-gthwh"]
Sep 11 17:49:05.051465 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:05.051429    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ee048b1-fc58-44cf-a1ec-2e45209ea089" path="/var/lib/kubelet/pods/5ee048b1-fc58-44cf-a1ec-2e45209ea089/volumes"
Sep 11 17:49:54.964515 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:49:54.964481    2559 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 17:50:06.764895 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:06.764854    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b"]
Sep 11 17:50:06.767279 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:06.765161    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ee048b1-fc58-44cf-a1ec-2e45209ea089" containerName="console"
Sep 11 17:50:06.767279 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:06.765172    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ee048b1-fc58-44cf-a1ec-2e45209ea089" containerName="console"
Sep 11 17:50:06.767279 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:06.765230    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ee048b1-fc58-44cf-a1ec-2e45209ea089" containerName="console"
Sep 11 17:50:06.768095 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:06.768081    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b"
Sep 11 17:50:06.771701 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:06.771671    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/744bd056-0760-4f91-b7f4-1cd885327be7-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b\" (UID: \"744bd056-0760-4f91-b7f4-1cd885327be7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b"
Sep 11 17:50:06.771810 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:06.771767    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/744bd056-0760-4f91-b7f4-1cd885327be7-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b\" (UID: \"744bd056-0760-4f91-b7f4-1cd885327be7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b"
Sep 11 17:50:06.771810 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:06.771797    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlqwb\" (UniqueName: \"kubernetes.io/projected/744bd056-0760-4f91-b7f4-1cd885327be7-kube-api-access-mlqwb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b\" (UID: \"744bd056-0760-4f91-b7f4-1cd885327be7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b"
Sep 11 17:50:06.771891 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:06.771837    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 17:50:06.772098 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:06.772084    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 17:50:06.772787 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:06.772768    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-n4v7b\""
Sep 11 17:50:06.782420 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:06.782400    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b"]
Sep 11 17:50:06.872460 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:06.872419    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/744bd056-0760-4f91-b7f4-1cd885327be7-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b\" (UID: \"744bd056-0760-4f91-b7f4-1cd885327be7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b"
Sep 11 17:50:06.872634 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:06.872501    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/744bd056-0760-4f91-b7f4-1cd885327be7-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b\" (UID: \"744bd056-0760-4f91-b7f4-1cd885327be7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b"
Sep 11 17:50:06.872634 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:06.872530    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlqwb\" (UniqueName: \"kubernetes.io/projected/744bd056-0760-4f91-b7f4-1cd885327be7-kube-api-access-mlqwb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b\" (UID: \"744bd056-0760-4f91-b7f4-1cd885327be7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b"
Sep 11 17:50:06.872817 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:06.872798    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/744bd056-0760-4f91-b7f4-1cd885327be7-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b\" (UID: \"744bd056-0760-4f91-b7f4-1cd885327be7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b"
Sep 11 17:50:06.872856 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:06.872829    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/744bd056-0760-4f91-b7f4-1cd885327be7-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b\" (UID: \"744bd056-0760-4f91-b7f4-1cd885327be7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b"
Sep 11 17:50:06.882122 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:06.882095    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlqwb\" (UniqueName: \"kubernetes.io/projected/744bd056-0760-4f91-b7f4-1cd885327be7-kube-api-access-mlqwb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b\" (UID: \"744bd056-0760-4f91-b7f4-1cd885327be7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b"
Sep 11 17:50:07.079488 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:07.079398    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b"
Sep 11 17:50:07.355235 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:07.355210    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b"]
Sep 11 17:50:07.357877 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:50:07.357848    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744bd056_0760_4f91_b7f4_1cd885327be7.slice/crio-220174383bbdcfb9500b7294a6d27777157379bec443505896c7b5121e15f47d WatchSource:0}: Error finding container 220174383bbdcfb9500b7294a6d27777157379bec443505896c7b5121e15f47d: Status 404 returned error can't find the container with id 220174383bbdcfb9500b7294a6d27777157379bec443505896c7b5121e15f47d
Sep 11 17:50:07.359853 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:07.359832    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:50:07.401411 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:07.401344    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b" event={"ID":"744bd056-0760-4f91-b7f4-1cd885327be7","Type":"ContainerStarted","Data":"220174383bbdcfb9500b7294a6d27777157379bec443505896c7b5121e15f47d"}
Sep 11 17:50:13.427871 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:13.427833    2559 generic.go:356] "Generic (PLEG): container finished" podID="744bd056-0760-4f91-b7f4-1cd885327be7" containerID="602a25fa6a98dbf3bfab8f741693ec1eda1e284855bdb593ec2e26813f489d41" exitCode=0
Sep 11 17:50:13.428407 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:13.427915    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b" event={"ID":"744bd056-0760-4f91-b7f4-1cd885327be7","Type":"ContainerDied","Data":"602a25fa6a98dbf3bfab8f741693ec1eda1e284855bdb593ec2e26813f489d41"}
Sep 11 17:50:15.438750 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:15.438716    2559 generic.go:356] "Generic (PLEG): container finished" podID="744bd056-0760-4f91-b7f4-1cd885327be7" containerID="a8b91f79d48212013b6234b54fb21e45ab947845b4bd5f2487d9e2fc69cdbe10" exitCode=0
Sep 11 17:50:15.439131 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:15.438804    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b" event={"ID":"744bd056-0760-4f91-b7f4-1cd885327be7","Type":"ContainerDied","Data":"a8b91f79d48212013b6234b54fb21e45ab947845b4bd5f2487d9e2fc69cdbe10"}
Sep 11 17:50:21.466588 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:21.466554    2559 generic.go:356] "Generic (PLEG): container finished" podID="744bd056-0760-4f91-b7f4-1cd885327be7" containerID="33f3ab951a4fd27df8f98cc3a1c2a9fa556dc5a6c0a96ce742eb470fe0ef1cfe" exitCode=0
Sep 11 17:50:21.467044 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:21.466634    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b" event={"ID":"744bd056-0760-4f91-b7f4-1cd885327be7","Type":"ContainerDied","Data":"33f3ab951a4fd27df8f98cc3a1c2a9fa556dc5a6c0a96ce742eb470fe0ef1cfe"}
Sep 11 17:50:22.597968 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:22.597944    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b"
Sep 11 17:50:22.718923 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:22.718872    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/744bd056-0760-4f91-b7f4-1cd885327be7-bundle\") pod \"744bd056-0760-4f91-b7f4-1cd885327be7\" (UID: \"744bd056-0760-4f91-b7f4-1cd885327be7\") "
Sep 11 17:50:22.718923 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:22.718922    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/744bd056-0760-4f91-b7f4-1cd885327be7-util\") pod \"744bd056-0760-4f91-b7f4-1cd885327be7\" (UID: \"744bd056-0760-4f91-b7f4-1cd885327be7\") "
Sep 11 17:50:22.719200 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:22.718946    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlqwb\" (UniqueName: \"kubernetes.io/projected/744bd056-0760-4f91-b7f4-1cd885327be7-kube-api-access-mlqwb\") pod \"744bd056-0760-4f91-b7f4-1cd885327be7\" (UID: \"744bd056-0760-4f91-b7f4-1cd885327be7\") "
Sep 11 17:50:22.719635 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:22.719606    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744bd056-0760-4f91-b7f4-1cd885327be7-bundle" (OuterVolumeSpecName: "bundle") pod "744bd056-0760-4f91-b7f4-1cd885327be7" (UID: "744bd056-0760-4f91-b7f4-1cd885327be7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:50:22.721203 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:22.721180    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744bd056-0760-4f91-b7f4-1cd885327be7-kube-api-access-mlqwb" (OuterVolumeSpecName: "kube-api-access-mlqwb") pod "744bd056-0760-4f91-b7f4-1cd885327be7" (UID: "744bd056-0760-4f91-b7f4-1cd885327be7"). InnerVolumeSpecName "kube-api-access-mlqwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:50:22.723408 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:22.723340    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744bd056-0760-4f91-b7f4-1cd885327be7-util" (OuterVolumeSpecName: "util") pod "744bd056-0760-4f91-b7f4-1cd885327be7" (UID: "744bd056-0760-4f91-b7f4-1cd885327be7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:50:22.820267 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:22.820215    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/744bd056-0760-4f91-b7f4-1cd885327be7-bundle\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:50:22.820267 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:22.820260    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/744bd056-0760-4f91-b7f4-1cd885327be7-util\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:50:22.820267 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:22.820270    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlqwb\" (UniqueName: \"kubernetes.io/projected/744bd056-0760-4f91-b7f4-1cd885327be7-kube-api-access-mlqwb\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:50:23.482151 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:23.482109    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b" event={"ID":"744bd056-0760-4f91-b7f4-1cd885327be7","Type":"ContainerDied","Data":"220174383bbdcfb9500b7294a6d27777157379bec443505896c7b5121e15f47d"}
Sep 11 17:50:23.482315 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:23.482182    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="220174383bbdcfb9500b7294a6d27777157379bec443505896c7b5121e15f47d"
Sep 11 17:50:23.482315 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:23.482127    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540x5n4b"
Sep 11 17:50:35.118070 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:35.118036    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss"]
Sep 11 17:50:35.118541 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:35.118377    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="744bd056-0760-4f91-b7f4-1cd885327be7" containerName="extract"
Sep 11 17:50:35.118541 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:35.118390    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="744bd056-0760-4f91-b7f4-1cd885327be7" containerName="extract"
Sep 11 17:50:35.118541 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:35.118409    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="744bd056-0760-4f91-b7f4-1cd885327be7" containerName="util"
Sep 11 17:50:35.118541 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:35.118415    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="744bd056-0760-4f91-b7f4-1cd885327be7" containerName="util"
Sep 11 17:50:35.118541 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:35.118422    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="744bd056-0760-4f91-b7f4-1cd885327be7" containerName="pull"
Sep 11 17:50:35.118541 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:35.118428    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="744bd056-0760-4f91-b7f4-1cd885327be7" containerName="pull"
Sep 11 17:50:35.118541 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:35.118481    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="744bd056-0760-4f91-b7f4-1cd885327be7" containerName="extract"
Sep 11 17:50:35.122320 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:35.122299    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss"
Sep 11 17:50:35.124796 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:35.124768    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 17:50:35.124937 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:35.124862    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 17:50:35.125710 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:35.125685    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-n4v7b\""
Sep 11 17:50:35.136749 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:35.136722    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss"]
Sep 11 17:50:35.234972 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:35.234932    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5flnj\" (UniqueName: \"kubernetes.io/projected/9513ffdb-3ab2-41e8-ba36-10321f5078d4-kube-api-access-5flnj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss\" (UID: \"9513ffdb-3ab2-41e8-ba36-10321f5078d4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss"
Sep 11 17:50:35.234972 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:35.234975    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9513ffdb-3ab2-41e8-ba36-10321f5078d4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss\" (UID: \"9513ffdb-3ab2-41e8-ba36-10321f5078d4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss"
Sep 11 17:50:35.235252 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:35.235046    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9513ffdb-3ab2-41e8-ba36-10321f5078d4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss\" (UID: \"9513ffdb-3ab2-41e8-ba36-10321f5078d4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss"
Sep 11 17:50:35.335614 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:35.335572    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5flnj\" (UniqueName: \"kubernetes.io/projected/9513ffdb-3ab2-41e8-ba36-10321f5078d4-kube-api-access-5flnj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss\" (UID: \"9513ffdb-3ab2-41e8-ba36-10321f5078d4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss"
Sep 11 17:50:35.335814 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:35.335620    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9513ffdb-3ab2-41e8-ba36-10321f5078d4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss\" (UID: \"9513ffdb-3ab2-41e8-ba36-10321f5078d4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss"
Sep 11 17:50:35.335814 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:35.335689    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9513ffdb-3ab2-41e8-ba36-10321f5078d4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss\" (UID: \"9513ffdb-3ab2-41e8-ba36-10321f5078d4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss"
Sep 11 17:50:35.336071 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:35.336046    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9513ffdb-3ab2-41e8-ba36-10321f5078d4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss\" (UID: \"9513ffdb-3ab2-41e8-ba36-10321f5078d4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss"
Sep 11 17:50:35.336140 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:35.336071    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9513ffdb-3ab2-41e8-ba36-10321f5078d4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss\" (UID: \"9513ffdb-3ab2-41e8-ba36-10321f5078d4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss"
Sep 11 17:50:35.364140 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:35.364103    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5flnj\" (UniqueName: \"kubernetes.io/projected/9513ffdb-3ab2-41e8-ba36-10321f5078d4-kube-api-access-5flnj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss\" (UID: \"9513ffdb-3ab2-41e8-ba36-10321f5078d4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss"
Sep 11 17:50:35.431341 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:35.431299    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss"
Sep 11 17:50:35.705837 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:35.705736    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss"]
Sep 11 17:50:35.708922 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:50:35.708884    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9513ffdb_3ab2_41e8_ba36_10321f5078d4.slice/crio-fc420bf4183cb71f6ec2fb5eeea099a8582a1e4d865beb23438150ac4eceaabe WatchSource:0}: Error finding container fc420bf4183cb71f6ec2fb5eeea099a8582a1e4d865beb23438150ac4eceaabe: Status 404 returned error can't find the container with id fc420bf4183cb71f6ec2fb5eeea099a8582a1e4d865beb23438150ac4eceaabe
Sep 11 17:50:36.533566 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:36.533528    2559 generic.go:356] "Generic (PLEG): container finished" podID="9513ffdb-3ab2-41e8-ba36-10321f5078d4" containerID="cf576e3e296284f0c08e6646eb702e2b70475f75dfc43c0a0d281e11c832ab8c" exitCode=0
Sep 11 17:50:36.534042 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:36.533612    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss" event={"ID":"9513ffdb-3ab2-41e8-ba36-10321f5078d4","Type":"ContainerDied","Data":"cf576e3e296284f0c08e6646eb702e2b70475f75dfc43c0a0d281e11c832ab8c"}
Sep 11 17:50:36.534042 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:36.533663    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss" event={"ID":"9513ffdb-3ab2-41e8-ba36-10321f5078d4","Type":"ContainerStarted","Data":"fc420bf4183cb71f6ec2fb5eeea099a8582a1e4d865beb23438150ac4eceaabe"}
Sep 11 17:50:39.501653 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:39.501613    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-nbqzm"]
Sep 11 17:50:39.504107 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:39.504086    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nbqzm"
Sep 11 17:50:39.506722 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:39.506694    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 17:50:39.506857 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:39.506694    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 17:50:39.507415 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:39.507396    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-npn9n\""
Sep 11 17:50:39.513818 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:39.513787    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-nbqzm"]
Sep 11 17:50:39.548434 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:39.548399    2559 generic.go:356] "Generic (PLEG): container finished" podID="9513ffdb-3ab2-41e8-ba36-10321f5078d4" containerID="25e864390036917cd527eed72e973dc51fbd90eec8be4f935327c02db0b2bbfb" exitCode=0
Sep 11 17:50:39.548645 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:39.548486    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss" event={"ID":"9513ffdb-3ab2-41e8-ba36-10321f5078d4","Type":"ContainerDied","Data":"25e864390036917cd527eed72e973dc51fbd90eec8be4f935327c02db0b2bbfb"}
Sep 11 17:50:39.675791 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:39.675750    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4x9l\" (UniqueName: \"kubernetes.io/projected/99710446-8839-4131-af10-acf4bd6c53d4-kube-api-access-s4x9l\") pod \"cert-manager-cainjector-559cdd8c87-nbqzm\" (UID: \"99710446-8839-4131-af10-acf4bd6c53d4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nbqzm"
Sep 11 17:50:39.675958 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:39.675830    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99710446-8839-4131-af10-acf4bd6c53d4-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-nbqzm\" (UID: \"99710446-8839-4131-af10-acf4bd6c53d4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nbqzm"
Sep 11 17:50:39.776941 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:39.776839    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4x9l\" (UniqueName: \"kubernetes.io/projected/99710446-8839-4131-af10-acf4bd6c53d4-kube-api-access-s4x9l\") pod \"cert-manager-cainjector-559cdd8c87-nbqzm\" (UID: \"99710446-8839-4131-af10-acf4bd6c53d4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nbqzm"
Sep 11 17:50:39.777109 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:39.776957    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99710446-8839-4131-af10-acf4bd6c53d4-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-nbqzm\" (UID: \"99710446-8839-4131-af10-acf4bd6c53d4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nbqzm"
Sep 11 17:50:39.786296 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:39.786266    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99710446-8839-4131-af10-acf4bd6c53d4-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-nbqzm\" (UID: \"99710446-8839-4131-af10-acf4bd6c53d4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nbqzm"
Sep 11 17:50:39.786529 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:39.786510    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4x9l\" (UniqueName: \"kubernetes.io/projected/99710446-8839-4131-af10-acf4bd6c53d4-kube-api-access-s4x9l\") pod \"cert-manager-cainjector-559cdd8c87-nbqzm\" (UID: \"99710446-8839-4131-af10-acf4bd6c53d4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-nbqzm"
Sep 11 17:50:39.822386 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:39.822325    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nbqzm"
Sep 11 17:50:40.091752 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:40.091628    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-nbqzm"]
Sep 11 17:50:40.094436 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:50:40.094408    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99710446_8839_4131_af10_acf4bd6c53d4.slice/crio-97230edb99aee2b27e9e1eb904a336a9627203d8196082418aad41291b0436a9 WatchSource:0}: Error finding container 97230edb99aee2b27e9e1eb904a336a9627203d8196082418aad41291b0436a9: Status 404 returned error can't find the container with id 97230edb99aee2b27e9e1eb904a336a9627203d8196082418aad41291b0436a9
Sep 11 17:50:40.560955 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:40.560915    2559 generic.go:356] "Generic (PLEG): container finished" podID="9513ffdb-3ab2-41e8-ba36-10321f5078d4" containerID="19d70324b3dbf0b8f03beacfa2ee7abe744d390ef724ddca9d104e95e4751399" exitCode=0
Sep 11 17:50:40.561441 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:40.561009    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss" event={"ID":"9513ffdb-3ab2-41e8-ba36-10321f5078d4","Type":"ContainerDied","Data":"19d70324b3dbf0b8f03beacfa2ee7abe744d390ef724ddca9d104e95e4751399"}
Sep 11 17:50:40.562364 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:40.562321    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nbqzm" event={"ID":"99710446-8839-4131-af10-acf4bd6c53d4","Type":"ContainerStarted","Data":"97230edb99aee2b27e9e1eb904a336a9627203d8196082418aad41291b0436a9"}
Sep 11 17:50:41.721532 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:41.721505    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss"
Sep 11 17:50:41.895819 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:41.895780    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5flnj\" (UniqueName: \"kubernetes.io/projected/9513ffdb-3ab2-41e8-ba36-10321f5078d4-kube-api-access-5flnj\") pod \"9513ffdb-3ab2-41e8-ba36-10321f5078d4\" (UID: \"9513ffdb-3ab2-41e8-ba36-10321f5078d4\") "
Sep 11 17:50:41.896030 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:41.895834    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9513ffdb-3ab2-41e8-ba36-10321f5078d4-util\") pod \"9513ffdb-3ab2-41e8-ba36-10321f5078d4\" (UID: \"9513ffdb-3ab2-41e8-ba36-10321f5078d4\") "
Sep 11 17:50:41.896030 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:41.895932    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9513ffdb-3ab2-41e8-ba36-10321f5078d4-bundle\") pod \"9513ffdb-3ab2-41e8-ba36-10321f5078d4\" (UID: \"9513ffdb-3ab2-41e8-ba36-10321f5078d4\") "
Sep 11 17:50:41.896409 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:41.896377    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9513ffdb-3ab2-41e8-ba36-10321f5078d4-bundle" (OuterVolumeSpecName: "bundle") pod "9513ffdb-3ab2-41e8-ba36-10321f5078d4" (UID: "9513ffdb-3ab2-41e8-ba36-10321f5078d4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:50:41.898406 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:41.898374    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9513ffdb-3ab2-41e8-ba36-10321f5078d4-kube-api-access-5flnj" (OuterVolumeSpecName: "kube-api-access-5flnj") pod "9513ffdb-3ab2-41e8-ba36-10321f5078d4" (UID: "9513ffdb-3ab2-41e8-ba36-10321f5078d4"). InnerVolumeSpecName "kube-api-access-5flnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:50:41.901535 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:41.901501    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9513ffdb-3ab2-41e8-ba36-10321f5078d4-util" (OuterVolumeSpecName: "util") pod "9513ffdb-3ab2-41e8-ba36-10321f5078d4" (UID: "9513ffdb-3ab2-41e8-ba36-10321f5078d4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:50:41.996759 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:41.996712    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5flnj\" (UniqueName: \"kubernetes.io/projected/9513ffdb-3ab2-41e8-ba36-10321f5078d4-kube-api-access-5flnj\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:50:41.996759 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:41.996757    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9513ffdb-3ab2-41e8-ba36-10321f5078d4-util\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:50:41.996759 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:41.996769    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9513ffdb-3ab2-41e8-ba36-10321f5078d4-bundle\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:50:42.574403 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:42.574344    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss" event={"ID":"9513ffdb-3ab2-41e8-ba36-10321f5078d4","Type":"ContainerDied","Data":"fc420bf4183cb71f6ec2fb5eeea099a8582a1e4d865beb23438150ac4eceaabe"}
Sep 11 17:50:42.574583 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:42.574436    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc420bf4183cb71f6ec2fb5eeea099a8582a1e4d865beb23438150ac4eceaabe"
Sep 11 17:50:42.574583 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:42.574372    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxjsss"
Sep 11 17:50:43.580119 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:43.580080    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nbqzm" event={"ID":"99710446-8839-4131-af10-acf4bd6c53d4","Type":"ContainerStarted","Data":"538921b8dfc7dab26b90b55c94697db96f826603da6d300d6d62a595942e470a"}
Sep 11 17:50:43.604189 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:43.604132    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-nbqzm" podStartSLOduration=2.054272818 podStartE2EDuration="4.604113361s" totalImagesPullingTime="2.549840543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:50:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:50:43.602935489 +0000 UTC m=+349.048036061" watchObservedRunningTime="2026-09-11 17:50:43.604113361 +0000 UTC m=+349.049213931"
Sep 11 17:50:55.655434 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:55.655329    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl"]
Sep 11 17:50:55.655826 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:55.655757    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9513ffdb-3ab2-41e8-ba36-10321f5078d4" containerName="extract"
Sep 11 17:50:55.655826 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:55.655772    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9513ffdb-3ab2-41e8-ba36-10321f5078d4" containerName="extract"
Sep 11 17:50:55.655826 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:55.655783    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9513ffdb-3ab2-41e8-ba36-10321f5078d4" containerName="util"
Sep 11 17:50:55.655826 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:55.655789    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9513ffdb-3ab2-41e8-ba36-10321f5078d4" containerName="util"
Sep 11 17:50:55.655826 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:55.655799    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9513ffdb-3ab2-41e8-ba36-10321f5078d4" containerName="pull"
Sep 11 17:50:55.655826 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:55.655805    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9513ffdb-3ab2-41e8-ba36-10321f5078d4" containerName="pull"
Sep 11 17:50:55.656029 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:55.655863    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9513ffdb-3ab2-41e8-ba36-10321f5078d4" containerName="extract"
Sep 11 17:50:55.657966 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:55.657949    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl"
Sep 11 17:50:55.660487 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:55.660458    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 17:50:55.660607 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:55.660539    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-n4v7b\""
Sep 11 17:50:55.660607 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:55.660589    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 17:50:55.667010 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:55.666981    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl"]
Sep 11 17:50:55.711494 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:55.711450    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8354082f-5320-41a9-aeb9-a310602ce234-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl\" (UID: \"8354082f-5320-41a9-aeb9-a310602ce234\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl"
Sep 11 17:50:55.711494 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:55.711499    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8354082f-5320-41a9-aeb9-a310602ce234-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl\" (UID: \"8354082f-5320-41a9-aeb9-a310602ce234\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl"
Sep 11 17:50:55.711708 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:55.711529    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhtb6\" (UniqueName: \"kubernetes.io/projected/8354082f-5320-41a9-aeb9-a310602ce234-kube-api-access-mhtb6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl\" (UID: \"8354082f-5320-41a9-aeb9-a310602ce234\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl"
Sep 11 17:50:55.811940 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:55.811904    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8354082f-5320-41a9-aeb9-a310602ce234-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl\" (UID: \"8354082f-5320-41a9-aeb9-a310602ce234\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl"
Sep 11 17:50:55.812112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:55.811953    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8354082f-5320-41a9-aeb9-a310602ce234-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl\" (UID: \"8354082f-5320-41a9-aeb9-a310602ce234\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl"
Sep 11 17:50:55.812112 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:55.811981    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhtb6\" (UniqueName: \"kubernetes.io/projected/8354082f-5320-41a9-aeb9-a310602ce234-kube-api-access-mhtb6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl\" (UID: \"8354082f-5320-41a9-aeb9-a310602ce234\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl"
Sep 11 17:50:55.812281 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:55.812263    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8354082f-5320-41a9-aeb9-a310602ce234-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl\" (UID: \"8354082f-5320-41a9-aeb9-a310602ce234\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl"
Sep 11 17:50:55.812421 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:55.812400    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8354082f-5320-41a9-aeb9-a310602ce234-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl\" (UID: \"8354082f-5320-41a9-aeb9-a310602ce234\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl"
Sep 11 17:50:55.820950 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:55.820916    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhtb6\" (UniqueName: \"kubernetes.io/projected/8354082f-5320-41a9-aeb9-a310602ce234-kube-api-access-mhtb6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl\" (UID: \"8354082f-5320-41a9-aeb9-a310602ce234\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl"
Sep 11 17:50:55.967491 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:55.967380    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl"
Sep 11 17:50:56.248161 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:56.248102    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl"]
Sep 11 17:50:56.250488 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:50:56.250459    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8354082f_5320_41a9_aeb9_a310602ce234.slice/crio-bd84a096cb790fcdb71d777bdd93e1e2f5c6e9f3d85829403af7ed99556cbb76 WatchSource:0}: Error finding container bd84a096cb790fcdb71d777bdd93e1e2f5c6e9f3d85829403af7ed99556cbb76: Status 404 returned error can't find the container with id bd84a096cb790fcdb71d777bdd93e1e2f5c6e9f3d85829403af7ed99556cbb76
Sep 11 17:50:56.637662 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:56.637626    2559 generic.go:356] "Generic (PLEG): container finished" podID="8354082f-5320-41a9-aeb9-a310602ce234" containerID="d20d61ece3e75c09d223d0751e7f96f80ef5b73f0f8060205c1c534040e17fee" exitCode=0
Sep 11 17:50:56.637880 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:56.637719    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl" event={"ID":"8354082f-5320-41a9-aeb9-a310602ce234","Type":"ContainerDied","Data":"d20d61ece3e75c09d223d0751e7f96f80ef5b73f0f8060205c1c534040e17fee"}
Sep 11 17:50:56.637880 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:56.637776    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl" event={"ID":"8354082f-5320-41a9-aeb9-a310602ce234","Type":"ContainerStarted","Data":"bd84a096cb790fcdb71d777bdd93e1e2f5c6e9f3d85829403af7ed99556cbb76"}
Sep 11 17:50:57.643958 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:57.643922    2559 generic.go:356] "Generic (PLEG): container finished" podID="8354082f-5320-41a9-aeb9-a310602ce234" containerID="455cecfd6d212c10c12c27163f4a4856d530c1c89caf337c307d8906a5199c30" exitCode=0
Sep 11 17:50:57.644398 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:57.643964    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl" event={"ID":"8354082f-5320-41a9-aeb9-a310602ce234","Type":"ContainerDied","Data":"455cecfd6d212c10c12c27163f4a4856d530c1c89caf337c307d8906a5199c30"}
Sep 11 17:50:58.653419 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:58.653372    2559 generic.go:356] "Generic (PLEG): container finished" podID="8354082f-5320-41a9-aeb9-a310602ce234" containerID="b93670a74b5294ffe0257c5d98416007b46d98933e7eb952385bd51f7465a615" exitCode=0
Sep 11 17:50:58.653834 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:58.653444    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl" event={"ID":"8354082f-5320-41a9-aeb9-a310602ce234","Type":"ContainerDied","Data":"b93670a74b5294ffe0257c5d98416007b46d98933e7eb952385bd51f7465a615"}
Sep 11 17:50:59.784364 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:59.784325    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl"
Sep 11 17:50:59.847566 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:59.847522    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8354082f-5320-41a9-aeb9-a310602ce234-bundle\") pod \"8354082f-5320-41a9-aeb9-a310602ce234\" (UID: \"8354082f-5320-41a9-aeb9-a310602ce234\") "
Sep 11 17:50:59.847736 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:59.847579    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8354082f-5320-41a9-aeb9-a310602ce234-util\") pod \"8354082f-5320-41a9-aeb9-a310602ce234\" (UID: \"8354082f-5320-41a9-aeb9-a310602ce234\") "
Sep 11 17:50:59.847736 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:59.847607    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhtb6\" (UniqueName: \"kubernetes.io/projected/8354082f-5320-41a9-aeb9-a310602ce234-kube-api-access-mhtb6\") pod \"8354082f-5320-41a9-aeb9-a310602ce234\" (UID: \"8354082f-5320-41a9-aeb9-a310602ce234\") "
Sep 11 17:50:59.848253 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:59.848213    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8354082f-5320-41a9-aeb9-a310602ce234-bundle" (OuterVolumeSpecName: "bundle") pod "8354082f-5320-41a9-aeb9-a310602ce234" (UID: "8354082f-5320-41a9-aeb9-a310602ce234"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:50:59.849721 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:59.849694    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8354082f-5320-41a9-aeb9-a310602ce234-kube-api-access-mhtb6" (OuterVolumeSpecName: "kube-api-access-mhtb6") pod "8354082f-5320-41a9-aeb9-a310602ce234" (UID: "8354082f-5320-41a9-aeb9-a310602ce234"). InnerVolumeSpecName "kube-api-access-mhtb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:50:59.853200 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:59.853159    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8354082f-5320-41a9-aeb9-a310602ce234-util" (OuterVolumeSpecName: "util") pod "8354082f-5320-41a9-aeb9-a310602ce234" (UID: "8354082f-5320-41a9-aeb9-a310602ce234"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:50:59.948901 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:59.948806    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8354082f-5320-41a9-aeb9-a310602ce234-bundle\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:50:59.948901 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:59.948842    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8354082f-5320-41a9-aeb9-a310602ce234-util\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:50:59.948901 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:50:59.948851    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhtb6\" (UniqueName: \"kubernetes.io/projected/8354082f-5320-41a9-aeb9-a310602ce234-kube-api-access-mhtb6\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:51:00.665080 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:00.665038    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl" event={"ID":"8354082f-5320-41a9-aeb9-a310602ce234","Type":"ContainerDied","Data":"bd84a096cb790fcdb71d777bdd93e1e2f5c6e9f3d85829403af7ed99556cbb76"}
Sep 11 17:51:00.665253 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:00.665099    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd84a096cb790fcdb71d777bdd93e1e2f5c6e9f3d85829403af7ed99556cbb76"
Sep 11 17:51:00.665253 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:00.665055    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8mwjl"
Sep 11 17:51:06.545195 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:06.545158    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4"]
Sep 11 17:51:06.545603 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:06.545485    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8354082f-5320-41a9-aeb9-a310602ce234" containerName="util"
Sep 11 17:51:06.545603 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:06.545499    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8354082f-5320-41a9-aeb9-a310602ce234" containerName="util"
Sep 11 17:51:06.545603 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:06.545506    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8354082f-5320-41a9-aeb9-a310602ce234" containerName="extract"
Sep 11 17:51:06.545603 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:06.545511    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8354082f-5320-41a9-aeb9-a310602ce234" containerName="extract"
Sep 11 17:51:06.545603 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:06.545527    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8354082f-5320-41a9-aeb9-a310602ce234" containerName="pull"
Sep 11 17:51:06.545603 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:06.545533    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8354082f-5320-41a9-aeb9-a310602ce234" containerName="pull"
Sep 11 17:51:06.545603 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:06.545587    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8354082f-5320-41a9-aeb9-a310602ce234" containerName="extract"
Sep 11 17:51:06.547717 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:06.547699    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4"
Sep 11 17:51:06.551584 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:06.551556    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 17:51:06.551709 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:06.551555    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-n4v7b\""
Sep 11 17:51:06.552390 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:06.552370    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 17:51:06.570543 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:06.570502    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4"]
Sep 11 17:51:06.604386 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:06.604328    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0966a53-ef0f-4f98-ac92-8cec6648844a-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4\" (UID: \"e0966a53-ef0f-4f98-ac92-8cec6648844a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4"
Sep 11 17:51:06.604566 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:06.604402    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0966a53-ef0f-4f98-ac92-8cec6648844a-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4\" (UID: \"e0966a53-ef0f-4f98-ac92-8cec6648844a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4"
Sep 11 17:51:06.604566 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:06.604441    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfmlr\" (UniqueName: \"kubernetes.io/projected/e0966a53-ef0f-4f98-ac92-8cec6648844a-kube-api-access-hfmlr\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4\" (UID: \"e0966a53-ef0f-4f98-ac92-8cec6648844a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4"
Sep 11 17:51:06.705482 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:06.705440    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfmlr\" (UniqueName: \"kubernetes.io/projected/e0966a53-ef0f-4f98-ac92-8cec6648844a-kube-api-access-hfmlr\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4\" (UID: \"e0966a53-ef0f-4f98-ac92-8cec6648844a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4"
Sep 11 17:51:06.705671 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:06.705544    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0966a53-ef0f-4f98-ac92-8cec6648844a-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4\" (UID: \"e0966a53-ef0f-4f98-ac92-8cec6648844a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4"
Sep 11 17:51:06.705671 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:06.705581    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0966a53-ef0f-4f98-ac92-8cec6648844a-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4\" (UID: \"e0966a53-ef0f-4f98-ac92-8cec6648844a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4"
Sep 11 17:51:06.706001 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:06.705981    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0966a53-ef0f-4f98-ac92-8cec6648844a-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4\" (UID: \"e0966a53-ef0f-4f98-ac92-8cec6648844a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4"
Sep 11 17:51:06.706041 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:06.705991    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0966a53-ef0f-4f98-ac92-8cec6648844a-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4\" (UID: \"e0966a53-ef0f-4f98-ac92-8cec6648844a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4"
Sep 11 17:51:06.715342 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:06.715316    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfmlr\" (UniqueName: \"kubernetes.io/projected/e0966a53-ef0f-4f98-ac92-8cec6648844a-kube-api-access-hfmlr\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4\" (UID: \"e0966a53-ef0f-4f98-ac92-8cec6648844a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4"
Sep 11 17:51:06.856841 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:06.856744    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4"
Sep 11 17:51:07.172254 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:07.172227    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4"]
Sep 11 17:51:07.174022 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:51:07.173993    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0966a53_ef0f_4f98_ac92_8cec6648844a.slice/crio-c912eb369c03d2bb6b0446d53b623a2e732034dc39f801705f09b1bcdf171b15 WatchSource:0}: Error finding container c912eb369c03d2bb6b0446d53b623a2e732034dc39f801705f09b1bcdf171b15: Status 404 returned error can't find the container with id c912eb369c03d2bb6b0446d53b623a2e732034dc39f801705f09b1bcdf171b15
Sep 11 17:51:07.695828 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:07.695793    2559 generic.go:356] "Generic (PLEG): container finished" podID="e0966a53-ef0f-4f98-ac92-8cec6648844a" containerID="491711d48bd2931592abe3850b02875cf9aa9eb74c6e5793f3e06afa2d930b1f" exitCode=0
Sep 11 17:51:07.696237 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:07.695888    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4" event={"ID":"e0966a53-ef0f-4f98-ac92-8cec6648844a","Type":"ContainerDied","Data":"491711d48bd2931592abe3850b02875cf9aa9eb74c6e5793f3e06afa2d930b1f"}
Sep 11 17:51:07.696237 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:07.695947    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4" event={"ID":"e0966a53-ef0f-4f98-ac92-8cec6648844a","Type":"ContainerStarted","Data":"c912eb369c03d2bb6b0446d53b623a2e732034dc39f801705f09b1bcdf171b15"}
Sep 11 17:51:08.702757 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.702660    2559 generic.go:356] "Generic (PLEG): container finished" podID="e0966a53-ef0f-4f98-ac92-8cec6648844a" containerID="ae947b3110de23c8db795af5cad5365fe4487e0da677c7547b1b9590c8fcf56b" exitCode=0
Sep 11 17:51:08.702757 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.702700    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4" event={"ID":"e0966a53-ef0f-4f98-ac92-8cec6648844a","Type":"ContainerDied","Data":"ae947b3110de23c8db795af5cad5365fe4487e0da677c7547b1b9590c8fcf56b"}
Sep 11 17:51:08.750282 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.750243    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs"]
Sep 11 17:51:08.755052 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.754954    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs"
Sep 11 17:51:08.758554 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.758523    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 17:51:08.758554 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.758532    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 17:51:08.758773 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.758557    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lgb85\""
Sep 11 17:51:08.758773 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.758712    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 17:51:08.759518 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.759500    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 17:51:08.779595 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.779562    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs"]
Sep 11 17:51:08.826864 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.826820    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f294e044-ccaf-4816-9149-c66d0cb26394-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f44bff486-v8pcs\" (UID: \"f294e044-ccaf-4816-9149-c66d0cb26394\") " pod="opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs"
Sep 11 17:51:08.827076 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.826935    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f294e044-ccaf-4816-9149-c66d0cb26394-tmp\") pod \"opendatahub-operator-controller-manager-6f44bff486-v8pcs\" (UID: \"f294e044-ccaf-4816-9149-c66d0cb26394\") " pod="opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs"
Sep 11 17:51:08.827076 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.826997    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4skxl\" (UniqueName: \"kubernetes.io/projected/f294e044-ccaf-4816-9149-c66d0cb26394-kube-api-access-4skxl\") pod \"opendatahub-operator-controller-manager-6f44bff486-v8pcs\" (UID: \"f294e044-ccaf-4816-9149-c66d0cb26394\") " pod="opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs"
Sep 11 17:51:08.827076 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.827022    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f294e044-ccaf-4816-9149-c66d0cb26394-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f44bff486-v8pcs\" (UID: \"f294e044-ccaf-4816-9149-c66d0cb26394\") " pod="opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs"
Sep 11 17:51:08.827256 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.827102    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f294e044-ccaf-4816-9149-c66d0cb26394-manifests\") pod \"opendatahub-operator-controller-manager-6f44bff486-v8pcs\" (UID: \"f294e044-ccaf-4816-9149-c66d0cb26394\") " pod="opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs"
Sep 11 17:51:08.928464 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.928421    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4skxl\" (UniqueName: \"kubernetes.io/projected/f294e044-ccaf-4816-9149-c66d0cb26394-kube-api-access-4skxl\") pod \"opendatahub-operator-controller-manager-6f44bff486-v8pcs\" (UID: \"f294e044-ccaf-4816-9149-c66d0cb26394\") " pod="opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs"
Sep 11 17:51:08.928464 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.928466    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f294e044-ccaf-4816-9149-c66d0cb26394-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f44bff486-v8pcs\" (UID: \"f294e044-ccaf-4816-9149-c66d0cb26394\") " pod="opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs"
Sep 11 17:51:08.928688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.928586    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f294e044-ccaf-4816-9149-c66d0cb26394-manifests\") pod \"opendatahub-operator-controller-manager-6f44bff486-v8pcs\" (UID: \"f294e044-ccaf-4816-9149-c66d0cb26394\") " pod="opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs"
Sep 11 17:51:08.928688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.928660    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f294e044-ccaf-4816-9149-c66d0cb26394-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f44bff486-v8pcs\" (UID: \"f294e044-ccaf-4816-9149-c66d0cb26394\") " pod="opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs"
Sep 11 17:51:08.928774 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.928711    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f294e044-ccaf-4816-9149-c66d0cb26394-tmp\") pod \"opendatahub-operator-controller-manager-6f44bff486-v8pcs\" (UID: \"f294e044-ccaf-4816-9149-c66d0cb26394\") " pod="opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs"
Sep 11 17:51:08.928997 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.928967    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f294e044-ccaf-4816-9149-c66d0cb26394-manifests\") pod \"opendatahub-operator-controller-manager-6f44bff486-v8pcs\" (UID: \"f294e044-ccaf-4816-9149-c66d0cb26394\") " pod="opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs"
Sep 11 17:51:08.930849 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.930816    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f294e044-ccaf-4816-9149-c66d0cb26394-tmp\") pod \"opendatahub-operator-controller-manager-6f44bff486-v8pcs\" (UID: \"f294e044-ccaf-4816-9149-c66d0cb26394\") " pod="opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs"
Sep 11 17:51:08.930958 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.930942    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f294e044-ccaf-4816-9149-c66d0cb26394-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f44bff486-v8pcs\" (UID: \"f294e044-ccaf-4816-9149-c66d0cb26394\") " pod="opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs"
Sep 11 17:51:08.931050 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.931033    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f294e044-ccaf-4816-9149-c66d0cb26394-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f44bff486-v8pcs\" (UID: \"f294e044-ccaf-4816-9149-c66d0cb26394\") " pod="opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs"
Sep 11 17:51:08.938269 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:08.938244    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4skxl\" (UniqueName: \"kubernetes.io/projected/f294e044-ccaf-4816-9149-c66d0cb26394-kube-api-access-4skxl\") pod \"opendatahub-operator-controller-manager-6f44bff486-v8pcs\" (UID: \"f294e044-ccaf-4816-9149-c66d0cb26394\") " pod="opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs"
Sep 11 17:51:09.070266 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:09.070174    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs"
Sep 11 17:51:09.360665 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:09.360578    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs"]
Sep 11 17:51:09.364607 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:51:09.364574    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf294e044_ccaf_4816_9149_c66d0cb26394.slice/crio-8fbfa825877c8e05777446fb5adb1b00c22ef36a00a0af95260513b090c5440b WatchSource:0}: Error finding container 8fbfa825877c8e05777446fb5adb1b00c22ef36a00a0af95260513b090c5440b: Status 404 returned error can't find the container with id 8fbfa825877c8e05777446fb5adb1b00c22ef36a00a0af95260513b090c5440b
Sep 11 17:51:09.709874 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:09.709840    2559 generic.go:356] "Generic (PLEG): container finished" podID="e0966a53-ef0f-4f98-ac92-8cec6648844a" containerID="d5da70c39975db87c839abf052be09969c55a18c096906b9b03518bd305f5eeb" exitCode=0
Sep 11 17:51:09.710307 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:09.709913    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4" event={"ID":"e0966a53-ef0f-4f98-ac92-8cec6648844a","Type":"ContainerDied","Data":"d5da70c39975db87c839abf052be09969c55a18c096906b9b03518bd305f5eeb"}
Sep 11 17:51:09.711220 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:09.711197    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs" event={"ID":"f294e044-ccaf-4816-9149-c66d0cb26394","Type":"ContainerStarted","Data":"8fbfa825877c8e05777446fb5adb1b00c22ef36a00a0af95260513b090c5440b"}
Sep 11 17:51:12.367433 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:12.367405    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4"
Sep 11 17:51:12.463894 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:12.463865    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0966a53-ef0f-4f98-ac92-8cec6648844a-util\") pod \"e0966a53-ef0f-4f98-ac92-8cec6648844a\" (UID: \"e0966a53-ef0f-4f98-ac92-8cec6648844a\") "
Sep 11 17:51:12.464073 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:12.463940    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfmlr\" (UniqueName: \"kubernetes.io/projected/e0966a53-ef0f-4f98-ac92-8cec6648844a-kube-api-access-hfmlr\") pod \"e0966a53-ef0f-4f98-ac92-8cec6648844a\" (UID: \"e0966a53-ef0f-4f98-ac92-8cec6648844a\") "
Sep 11 17:51:12.464073 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:12.463974    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0966a53-ef0f-4f98-ac92-8cec6648844a-bundle\") pod \"e0966a53-ef0f-4f98-ac92-8cec6648844a\" (UID: \"e0966a53-ef0f-4f98-ac92-8cec6648844a\") "
Sep 11 17:51:12.464713 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:12.464687    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0966a53-ef0f-4f98-ac92-8cec6648844a-bundle" (OuterVolumeSpecName: "bundle") pod "e0966a53-ef0f-4f98-ac92-8cec6648844a" (UID: "e0966a53-ef0f-4f98-ac92-8cec6648844a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:51:12.466217 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:12.466185    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0966a53-ef0f-4f98-ac92-8cec6648844a-kube-api-access-hfmlr" (OuterVolumeSpecName: "kube-api-access-hfmlr") pod "e0966a53-ef0f-4f98-ac92-8cec6648844a" (UID: "e0966a53-ef0f-4f98-ac92-8cec6648844a"). InnerVolumeSpecName "kube-api-access-hfmlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:51:12.469538 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:12.469513    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0966a53-ef0f-4f98-ac92-8cec6648844a-util" (OuterVolumeSpecName: "util") pod "e0966a53-ef0f-4f98-ac92-8cec6648844a" (UID: "e0966a53-ef0f-4f98-ac92-8cec6648844a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:51:12.565084 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:12.565051    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0966a53-ef0f-4f98-ac92-8cec6648844a-bundle\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:51:12.565084 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:12.565079    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0966a53-ef0f-4f98-ac92-8cec6648844a-util\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:51:12.565084 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:12.565089    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfmlr\" (UniqueName: \"kubernetes.io/projected/e0966a53-ef0f-4f98-ac92-8cec6648844a-kube-api-access-hfmlr\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:51:12.729315 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:12.729276    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4" event={"ID":"e0966a53-ef0f-4f98-ac92-8cec6648844a","Type":"ContainerDied","Data":"c912eb369c03d2bb6b0446d53b623a2e732034dc39f801705f09b1bcdf171b15"}
Sep 11 17:51:12.729552 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:12.729338    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c912eb369c03d2bb6b0446d53b623a2e732034dc39f801705f09b1bcdf171b15"
Sep 11 17:51:12.729552 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:12.729293    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0n65p4"
Sep 11 17:51:12.731176 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:12.731136    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs" event={"ID":"f294e044-ccaf-4816-9149-c66d0cb26394","Type":"ContainerStarted","Data":"96b69f6b0c4168d04539c543229baf420dcea764e609a3b603ea55e52c90057f"}
Sep 11 17:51:13.737742 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:13.737705    2559 generic.go:356] "Generic (PLEG): container finished" podID="f294e044-ccaf-4816-9149-c66d0cb26394" containerID="96b69f6b0c4168d04539c543229baf420dcea764e609a3b603ea55e52c90057f" exitCode=0
Sep 11 17:51:13.738171 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:13.737749    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs" event={"ID":"f294e044-ccaf-4816-9149-c66d0cb26394","Type":"ContainerDied","Data":"96b69f6b0c4168d04539c543229baf420dcea764e609a3b603ea55e52c90057f"}
Sep 11 17:51:14.744666 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:14.744625    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs" event={"ID":"f294e044-ccaf-4816-9149-c66d0cb26394","Type":"ContainerStarted","Data":"5b1ca6dedbeea0e2abab8a3370c41e18077324dce66915fdb8e2a3946db40e12"}
Sep 11 17:51:14.745164 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:14.744794    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs"
Sep 11 17:51:14.769267 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:14.769213    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs" podStartSLOduration=2.152447659 podStartE2EDuration="6.769196166s" totalImagesPullingTime="3.445940971s" totalInitContainerRuntime="1.170807536s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:51:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:51:14.767557117 +0000 UTC m=+380.212657687" watchObservedRunningTime="2026-09-11 17:51:14.769196166 +0000 UTC m=+380.214296738"
Sep 11 17:51:25.751874 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:25.751842    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6f44bff486-v8pcs"
Sep 11 17:51:26.150405 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.150346    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-f67fb7585-lmq4h"]
Sep 11 17:51:26.150726 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.150712    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0966a53-ef0f-4f98-ac92-8cec6648844a" containerName="pull"
Sep 11 17:51:26.150776 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.150727    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0966a53-ef0f-4f98-ac92-8cec6648844a" containerName="pull"
Sep 11 17:51:26.150776 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.150738    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0966a53-ef0f-4f98-ac92-8cec6648844a" containerName="util"
Sep 11 17:51:26.150776 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.150743    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0966a53-ef0f-4f98-ac92-8cec6648844a" containerName="util"
Sep 11 17:51:26.150776 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.150751    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0966a53-ef0f-4f98-ac92-8cec6648844a" containerName="extract"
Sep 11 17:51:26.150776 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.150756    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0966a53-ef0f-4f98-ac92-8cec6648844a" containerName="extract"
Sep 11 17:51:26.150919 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.150815    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0966a53-ef0f-4f98-ac92-8cec6648844a" containerName="extract"
Sep 11 17:51:26.157897 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.157865    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-f67fb7585-lmq4h"
Sep 11 17:51:26.165223 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.165200    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 17:51:26.165477 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.165462    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 17:51:26.165566 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.165477    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 17:51:26.165694 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.165681    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 17:51:26.166042 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.166025    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:51:26.166123 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.166046    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-lj67d\""
Sep 11 17:51:26.198374 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.198330    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-f67fb7585-lmq4h"]
Sep 11 17:51:26.286021 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.285983    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/124e9246-3163-48ed-a3e5-a55fc414b38a-metrics-cert\") pod \"lws-controller-manager-f67fb7585-lmq4h\" (UID: \"124e9246-3163-48ed-a3e5-a55fc414b38a\") " pod="openshift-lws-operator/lws-controller-manager-f67fb7585-lmq4h"
Sep 11 17:51:26.286021 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.286028    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bct4f\" (UniqueName: \"kubernetes.io/projected/124e9246-3163-48ed-a3e5-a55fc414b38a-kube-api-access-bct4f\") pod \"lws-controller-manager-f67fb7585-lmq4h\" (UID: \"124e9246-3163-48ed-a3e5-a55fc414b38a\") " pod="openshift-lws-operator/lws-controller-manager-f67fb7585-lmq4h"
Sep 11 17:51:26.286248 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.286138    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/124e9246-3163-48ed-a3e5-a55fc414b38a-cert\") pod \"lws-controller-manager-f67fb7585-lmq4h\" (UID: \"124e9246-3163-48ed-a3e5-a55fc414b38a\") " pod="openshift-lws-operator/lws-controller-manager-f67fb7585-lmq4h"
Sep 11 17:51:26.286248 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.286207    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/124e9246-3163-48ed-a3e5-a55fc414b38a-manager-config\") pod \"lws-controller-manager-f67fb7585-lmq4h\" (UID: \"124e9246-3163-48ed-a3e5-a55fc414b38a\") " pod="openshift-lws-operator/lws-controller-manager-f67fb7585-lmq4h"
Sep 11 17:51:26.387246 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.387198    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/124e9246-3163-48ed-a3e5-a55fc414b38a-metrics-cert\") pod \"lws-controller-manager-f67fb7585-lmq4h\" (UID: \"124e9246-3163-48ed-a3e5-a55fc414b38a\") " pod="openshift-lws-operator/lws-controller-manager-f67fb7585-lmq4h"
Sep 11 17:51:26.387486 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.387263    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bct4f\" (UniqueName: \"kubernetes.io/projected/124e9246-3163-48ed-a3e5-a55fc414b38a-kube-api-access-bct4f\") pod \"lws-controller-manager-f67fb7585-lmq4h\" (UID: \"124e9246-3163-48ed-a3e5-a55fc414b38a\") " pod="openshift-lws-operator/lws-controller-manager-f67fb7585-lmq4h"
Sep 11 17:51:26.387486 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.387399    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/124e9246-3163-48ed-a3e5-a55fc414b38a-cert\") pod \"lws-controller-manager-f67fb7585-lmq4h\" (UID: \"124e9246-3163-48ed-a3e5-a55fc414b38a\") " pod="openshift-lws-operator/lws-controller-manager-f67fb7585-lmq4h"
Sep 11 17:51:26.387486 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.387450    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/124e9246-3163-48ed-a3e5-a55fc414b38a-manager-config\") pod \"lws-controller-manager-f67fb7585-lmq4h\" (UID: \"124e9246-3163-48ed-a3e5-a55fc414b38a\") " pod="openshift-lws-operator/lws-controller-manager-f67fb7585-lmq4h"
Sep 11 17:51:26.388147 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.388117    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/124e9246-3163-48ed-a3e5-a55fc414b38a-manager-config\") pod \"lws-controller-manager-f67fb7585-lmq4h\" (UID: \"124e9246-3163-48ed-a3e5-a55fc414b38a\") " pod="openshift-lws-operator/lws-controller-manager-f67fb7585-lmq4h"
Sep 11 17:51:26.389898 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.389865    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/124e9246-3163-48ed-a3e5-a55fc414b38a-cert\") pod \"lws-controller-manager-f67fb7585-lmq4h\" (UID: \"124e9246-3163-48ed-a3e5-a55fc414b38a\") " pod="openshift-lws-operator/lws-controller-manager-f67fb7585-lmq4h"
Sep 11 17:51:26.390010 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.389949    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/124e9246-3163-48ed-a3e5-a55fc414b38a-metrics-cert\") pod \"lws-controller-manager-f67fb7585-lmq4h\" (UID: \"124e9246-3163-48ed-a3e5-a55fc414b38a\") " pod="openshift-lws-operator/lws-controller-manager-f67fb7585-lmq4h"
Sep 11 17:51:26.410150 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.410068    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bct4f\" (UniqueName: \"kubernetes.io/projected/124e9246-3163-48ed-a3e5-a55fc414b38a-kube-api-access-bct4f\") pod \"lws-controller-manager-f67fb7585-lmq4h\" (UID: \"124e9246-3163-48ed-a3e5-a55fc414b38a\") " pod="openshift-lws-operator/lws-controller-manager-f67fb7585-lmq4h"
Sep 11 17:51:26.467043 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.466987    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-f67fb7585-lmq4h"
Sep 11 17:51:26.754644 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.754493    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-f67fb7585-lmq4h"]
Sep 11 17:51:26.757413 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:51:26.757384    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124e9246_3163_48ed_a3e5_a55fc414b38a.slice/crio-22eb733353d97e7cf0ebebbda321d594d3fb55732f0eae7c08909c3c5ff86ea5 WatchSource:0}: Error finding container 22eb733353d97e7cf0ebebbda321d594d3fb55732f0eae7c08909c3c5ff86ea5: Status 404 returned error can't find the container with id 22eb733353d97e7cf0ebebbda321d594d3fb55732f0eae7c08909c3c5ff86ea5
Sep 11 17:51:26.800647 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:26.800609    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-f67fb7585-lmq4h" event={"ID":"124e9246-3163-48ed-a3e5-a55fc414b38a","Type":"ContainerStarted","Data":"22eb733353d97e7cf0ebebbda321d594d3fb55732f0eae7c08909c3c5ff86ea5"}
Sep 11 17:51:28.813039 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:28.813002    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-f67fb7585-lmq4h" event={"ID":"124e9246-3163-48ed-a3e5-a55fc414b38a","Type":"ContainerStarted","Data":"244c775d528ad9316e7261fd19f89e476ad857b7ba09ee206b22a92a6a0b1be7"}
Sep 11 17:51:28.813459 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:28.813115    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-f67fb7585-lmq4h"
Sep 11 17:51:28.891095 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:28.891029    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-f67fb7585-lmq4h" podStartSLOduration=1.3561157050000001 podStartE2EDuration="2.891008049s" totalImagesPullingTime="1.534892344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:51:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:51:28.888008012 +0000 UTC m=+394.333108582" watchObservedRunningTime="2026-09-11 17:51:28.891008049 +0000 UTC m=+394.336108620"
Sep 11 17:51:29.762653 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:29.762613    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-77c49cb9c9-4kwsl"]
Sep 11 17:51:29.765898 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:29.765877    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77c49cb9c9-4kwsl"
Sep 11 17:51:29.770600 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:29.770569    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 17:51:29.770740 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:29.770576    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9h5zb\""
Sep 11 17:51:29.773495 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:29.773476    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 17:51:29.805900 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:29.805863    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77c49cb9c9-4kwsl"]
Sep 11 17:51:29.921261 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:29.921218    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15500cdb-e686-4698-97fb-a0d0c5756a17-tmp\") pod \"kube-auth-proxy-77c49cb9c9-4kwsl\" (UID: \"15500cdb-e686-4698-97fb-a0d0c5756a17\") " pod="openshift-ingress/kube-auth-proxy-77c49cb9c9-4kwsl"
Sep 11 17:51:29.921742 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:29.921267    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b66w5\" (UniqueName: \"kubernetes.io/projected/15500cdb-e686-4698-97fb-a0d0c5756a17-kube-api-access-b66w5\") pod \"kube-auth-proxy-77c49cb9c9-4kwsl\" (UID: \"15500cdb-e686-4698-97fb-a0d0c5756a17\") " pod="openshift-ingress/kube-auth-proxy-77c49cb9c9-4kwsl"
Sep 11 17:51:29.921742 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:29.921380    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15500cdb-e686-4698-97fb-a0d0c5756a17-tls-certs\") pod \"kube-auth-proxy-77c49cb9c9-4kwsl\" (UID: \"15500cdb-e686-4698-97fb-a0d0c5756a17\") " pod="openshift-ingress/kube-auth-proxy-77c49cb9c9-4kwsl"
Sep 11 17:51:30.022246 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:30.022142    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15500cdb-e686-4698-97fb-a0d0c5756a17-tmp\") pod \"kube-auth-proxy-77c49cb9c9-4kwsl\" (UID: \"15500cdb-e686-4698-97fb-a0d0c5756a17\") " pod="openshift-ingress/kube-auth-proxy-77c49cb9c9-4kwsl"
Sep 11 17:51:30.022246 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:30.022191    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b66w5\" (UniqueName: \"kubernetes.io/projected/15500cdb-e686-4698-97fb-a0d0c5756a17-kube-api-access-b66w5\") pod \"kube-auth-proxy-77c49cb9c9-4kwsl\" (UID: \"15500cdb-e686-4698-97fb-a0d0c5756a17\") " pod="openshift-ingress/kube-auth-proxy-77c49cb9c9-4kwsl"
Sep 11 17:51:30.022246 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:30.022229    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15500cdb-e686-4698-97fb-a0d0c5756a17-tls-certs\") pod \"kube-auth-proxy-77c49cb9c9-4kwsl\" (UID: \"15500cdb-e686-4698-97fb-a0d0c5756a17\") " pod="openshift-ingress/kube-auth-proxy-77c49cb9c9-4kwsl"
Sep 11 17:51:30.022551 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:51:30.022404    2559 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 11 17:51:30.022551 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:51:30.022460    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15500cdb-e686-4698-97fb-a0d0c5756a17-tls-certs podName:15500cdb-e686-4698-97fb-a0d0c5756a17 nodeName:}" failed. No retries permitted until 2026-09-11 17:51:30.522444611 +0000 UTC m=+395.967545164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/15500cdb-e686-4698-97fb-a0d0c5756a17-tls-certs") pod "kube-auth-proxy-77c49cb9c9-4kwsl" (UID: "15500cdb-e686-4698-97fb-a0d0c5756a17") : secret "kube-auth-proxy-tls" not found
Sep 11 17:51:30.024511 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:30.024489    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15500cdb-e686-4698-97fb-a0d0c5756a17-tmp\") pod \"kube-auth-proxy-77c49cb9c9-4kwsl\" (UID: \"15500cdb-e686-4698-97fb-a0d0c5756a17\") " pod="openshift-ingress/kube-auth-proxy-77c49cb9c9-4kwsl"
Sep 11 17:51:30.073851 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:30.073817    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b66w5\" (UniqueName: \"kubernetes.io/projected/15500cdb-e686-4698-97fb-a0d0c5756a17-kube-api-access-b66w5\") pod \"kube-auth-proxy-77c49cb9c9-4kwsl\" (UID: \"15500cdb-e686-4698-97fb-a0d0c5756a17\") " pod="openshift-ingress/kube-auth-proxy-77c49cb9c9-4kwsl"
Sep 11 17:51:30.527382 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:30.527306    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15500cdb-e686-4698-97fb-a0d0c5756a17-tls-certs\") pod \"kube-auth-proxy-77c49cb9c9-4kwsl\" (UID: \"15500cdb-e686-4698-97fb-a0d0c5756a17\") " pod="openshift-ingress/kube-auth-proxy-77c49cb9c9-4kwsl"
Sep 11 17:51:30.529846 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:30.529817    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15500cdb-e686-4698-97fb-a0d0c5756a17-tls-certs\") pod \"kube-auth-proxy-77c49cb9c9-4kwsl\" (UID: \"15500cdb-e686-4698-97fb-a0d0c5756a17\") " pod="openshift-ingress/kube-auth-proxy-77c49cb9c9-4kwsl"
Sep 11 17:51:30.675820 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:30.675779    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77c49cb9c9-4kwsl"
Sep 11 17:51:30.960420 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:30.960393    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77c49cb9c9-4kwsl"]
Sep 11 17:51:30.961930 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:51:30.961891    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15500cdb_e686_4698_97fb_a0d0c5756a17.slice/crio-6ec85ca39d20a6a5e2fe4508c981df0de07b49f67313e1afd09478ce4b065f8c WatchSource:0}: Error finding container 6ec85ca39d20a6a5e2fe4508c981df0de07b49f67313e1afd09478ce4b065f8c: Status 404 returned error can't find the container with id 6ec85ca39d20a6a5e2fe4508c981df0de07b49f67313e1afd09478ce4b065f8c
Sep 11 17:51:31.842307 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:31.842259    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77c49cb9c9-4kwsl" event={"ID":"15500cdb-e686-4698-97fb-a0d0c5756a17","Type":"ContainerStarted","Data":"6ec85ca39d20a6a5e2fe4508c981df0de07b49f67313e1afd09478ce4b065f8c"}
Sep 11 17:51:34.858689 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:34.858650    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77c49cb9c9-4kwsl" event={"ID":"15500cdb-e686-4698-97fb-a0d0c5756a17","Type":"ContainerStarted","Data":"2cdab03146fd972ac596de36a708af663a706fcbf92a76e73465187b64d15dab"}
Sep 11 17:51:34.876436 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:34.876380    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-77c49cb9c9-4kwsl" podStartSLOduration=3.008009878 podStartE2EDuration="5.876362627s" totalImagesPullingTime="2.868352749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:51:34.874952766 +0000 UTC m=+400.320053337" watchObservedRunningTime="2026-09-11 17:51:34.876362627 +0000 UTC m=+400.321463402"
Sep 11 17:51:39.820588 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:39.820553    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-f67fb7585-lmq4h"
Sep 11 17:51:58.619795 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:51:58.619752    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d7c75f5d6-7xbxm"]
Sep 11 17:52:23.638924 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:23.638823    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d7c75f5d6-7xbxm" podUID="eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd" containerName="console" containerID="cri-o://fe7e3b7af966a8021e9339be1e90d069c5061e093b9bf9c2f951f29579bf5ddf" gracePeriod=15
Sep 11 17:52:23.892483 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:23.892419    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:52:23.998697 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:23.998656    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-console-serving-cert\") pod \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") "
Sep 11 17:52:23.998898 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:23.998707    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-console-oauth-config\") pod \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") "
Sep 11 17:52:23.998898 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:23.998815    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-console-config\") pod \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") "
Sep 11 17:52:23.998898 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:23.998845    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-service-ca\") pod \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") "
Sep 11 17:52:23.998898 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:23.998888    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnl46\" (UniqueName: \"kubernetes.io/projected/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-kube-api-access-vnl46\") pod \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") "
Sep 11 17:52:23.999116 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:23.998949    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-trusted-ca-bundle\") pod \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") "
Sep 11 17:52:23.999116 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:23.998991    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-oauth-serving-cert\") pod \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\" (UID: \"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd\") "
Sep 11 17:52:23.999262 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:23.999237    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-console-config" (OuterVolumeSpecName: "console-config") pod "eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd" (UID: "eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:52:23.999316 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:23.999255    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-service-ca" (OuterVolumeSpecName: "service-ca") pod "eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd" (UID: "eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:52:23.999396 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:23.999317    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd" (UID: "eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:52:23.999614 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:23.999590    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd" (UID: "eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:52:24.001029 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:24.001001    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd" (UID: "eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:52:24.001122 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:24.001046    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd" (UID: "eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:52:24.001122 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:24.001052    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-kube-api-access-vnl46" (OuterVolumeSpecName: "kube-api-access-vnl46") pod "eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd" (UID: "eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd"). InnerVolumeSpecName "kube-api-access-vnl46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:52:24.092311 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:24.092278    2559 generic.go:356] "Generic (PLEG): container finished" podID="eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd" containerID="fe7e3b7af966a8021e9339be1e90d069c5061e093b9bf9c2f951f29579bf5ddf" exitCode=0
Sep 11 17:52:24.092502 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:24.092327    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d7c75f5d6-7xbxm"
Sep 11 17:52:24.092502 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:24.092392    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d7c75f5d6-7xbxm" event={"ID":"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd","Type":"ContainerDied","Data":"fe7e3b7af966a8021e9339be1e90d069c5061e093b9bf9c2f951f29579bf5ddf"}
Sep 11 17:52:24.092502 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:24.092439    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d7c75f5d6-7xbxm" event={"ID":"eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd","Type":"ContainerDied","Data":"f541c825e4cd1ed21e15ec1e6bb8b51b00e2ae3954e68c9370c78bc3f3dd3ca4"}
Sep 11 17:52:24.092502 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:24.092465    2559 scope.go:117] "RemoveContainer" containerID="fe7e3b7af966a8021e9339be1e90d069c5061e093b9bf9c2f951f29579bf5ddf"
Sep 11 17:52:24.099643 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:24.099621    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-console-oauth-config\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:52:24.099643 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:24.099643    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-console-config\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:52:24.099777 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:24.099652    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-service-ca\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:52:24.099777 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:24.099660    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnl46\" (UniqueName: \"kubernetes.io/projected/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-kube-api-access-vnl46\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:52:24.099777 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:24.099668    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-trusted-ca-bundle\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:52:24.099777 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:24.099679    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-oauth-serving-cert\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:52:24.099777 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:24.099691    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd-console-serving-cert\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:52:24.102041 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:24.102026    2559 scope.go:117] "RemoveContainer" containerID="fe7e3b7af966a8021e9339be1e90d069c5061e093b9bf9c2f951f29579bf5ddf"
Sep 11 17:52:24.102277 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:52:24.102255    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe7e3b7af966a8021e9339be1e90d069c5061e093b9bf9c2f951f29579bf5ddf\": container with ID starting with fe7e3b7af966a8021e9339be1e90d069c5061e093b9bf9c2f951f29579bf5ddf not found: ID does not exist" containerID="fe7e3b7af966a8021e9339be1e90d069c5061e093b9bf9c2f951f29579bf5ddf"
Sep 11 17:52:24.102395 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:24.102281    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7e3b7af966a8021e9339be1e90d069c5061e093b9bf9c2f951f29579bf5ddf"} err="failed to get container status \"fe7e3b7af966a8021e9339be1e90d069c5061e093b9bf9c2f951f29579bf5ddf\": rpc error: code = NotFound desc = could not find container \"fe7e3b7af966a8021e9339be1e90d069c5061e093b9bf9c2f951f29579bf5ddf\": container with ID starting with fe7e3b7af966a8021e9339be1e90d069c5061e093b9bf9c2f951f29579bf5ddf not found: ID does not exist"
Sep 11 17:52:24.118968 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:24.118937    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d7c75f5d6-7xbxm"]
Sep 11 17:52:24.124145 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:24.124121    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d7c75f5d6-7xbxm"]
Sep 11 17:52:25.051800 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:25.051768    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd" path="/var/lib/kubelet/pods/eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd/volumes"
Sep 11 17:52:46.349071 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.349032    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj"]
Sep 11 17:52:46.349582 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.349513    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd" containerName="console"
Sep 11 17:52:46.349582 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.349534    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd" containerName="console"
Sep 11 17:52:46.349694 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.349677    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb8ce2d4-dbbb-4189-ba9d-94c6363efcdd" containerName="console"
Sep 11 17:52:46.354365 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.354331    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj"
Sep 11 17:52:46.356697 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.356671    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 17:52:46.356697 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.356671    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 17:52:46.357449 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.357428    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-n4v7b\""
Sep 11 17:52:46.362498 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.362474    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj"]
Sep 11 17:52:46.387084 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.387054    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbn7r\" (UniqueName: \"kubernetes.io/projected/cf22167a-8633-431b-a107-5f2febb8c17b-kube-api-access-dbn7r\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj\" (UID: \"cf22167a-8633-431b-a107-5f2febb8c17b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj"
Sep 11 17:52:46.387248 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.387109    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf22167a-8633-431b-a107-5f2febb8c17b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj\" (UID: \"cf22167a-8633-431b-a107-5f2febb8c17b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj"
Sep 11 17:52:46.387248 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.387163    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf22167a-8633-431b-a107-5f2febb8c17b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj\" (UID: \"cf22167a-8633-431b-a107-5f2febb8c17b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj"
Sep 11 17:52:46.454005 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.453968    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2"]
Sep 11 17:52:46.457246 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.457229    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2"
Sep 11 17:52:46.466971 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.466947    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2"]
Sep 11 17:52:46.487876 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.487848    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvvbw\" (UniqueName: \"kubernetes.io/projected/bd9404a7-c012-4078-8400-b4414ef2e8f1-kube-api-access-mvvbw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2\" (UID: \"bd9404a7-c012-4078-8400-b4414ef2e8f1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2"
Sep 11 17:52:46.488004 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.487897    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf22167a-8633-431b-a107-5f2febb8c17b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj\" (UID: \"cf22167a-8633-431b-a107-5f2febb8c17b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj"
Sep 11 17:52:46.488004 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.487972    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd9404a7-c012-4078-8400-b4414ef2e8f1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2\" (UID: \"bd9404a7-c012-4078-8400-b4414ef2e8f1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2"
Sep 11 17:52:46.488084 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.488038    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf22167a-8633-431b-a107-5f2febb8c17b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj\" (UID: \"cf22167a-8633-431b-a107-5f2febb8c17b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj"
Sep 11 17:52:46.488128 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.488087    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbn7r\" (UniqueName: \"kubernetes.io/projected/cf22167a-8633-431b-a107-5f2febb8c17b-kube-api-access-dbn7r\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj\" (UID: \"cf22167a-8633-431b-a107-5f2febb8c17b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj"
Sep 11 17:52:46.488128 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.488114    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd9404a7-c012-4078-8400-b4414ef2e8f1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2\" (UID: \"bd9404a7-c012-4078-8400-b4414ef2e8f1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2"
Sep 11 17:52:46.488268 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.488251    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf22167a-8633-431b-a107-5f2febb8c17b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj\" (UID: \"cf22167a-8633-431b-a107-5f2febb8c17b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj"
Sep 11 17:52:46.488392 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.488344    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf22167a-8633-431b-a107-5f2febb8c17b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj\" (UID: \"cf22167a-8633-431b-a107-5f2febb8c17b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj"
Sep 11 17:52:46.496209 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.496181    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbn7r\" (UniqueName: \"kubernetes.io/projected/cf22167a-8633-431b-a107-5f2febb8c17b-kube-api-access-dbn7r\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj\" (UID: \"cf22167a-8633-431b-a107-5f2febb8c17b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj"
Sep 11 17:52:46.546836 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.546802    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t"]
Sep 11 17:52:46.550155 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.550139    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t"
Sep 11 17:52:46.559377 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.559334    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t"]
Sep 11 17:52:46.589053 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.589024    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd9404a7-c012-4078-8400-b4414ef2e8f1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2\" (UID: \"bd9404a7-c012-4078-8400-b4414ef2e8f1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2"
Sep 11 17:52:46.589193 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.589089    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2aa7990-2e58-46f5-b596-1865212e0e39-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t\" (UID: \"f2aa7990-2e58-46f5-b596-1865212e0e39\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t"
Sep 11 17:52:46.589193 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.589112    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd9404a7-c012-4078-8400-b4414ef2e8f1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2\" (UID: \"bd9404a7-c012-4078-8400-b4414ef2e8f1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2"
Sep 11 17:52:46.589193 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.589142    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvx4v\" (UniqueName: \"kubernetes.io/projected/f2aa7990-2e58-46f5-b596-1865212e0e39-kube-api-access-mvx4v\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t\" (UID: \"f2aa7990-2e58-46f5-b596-1865212e0e39\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t"
Sep 11 17:52:46.589193 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.589170    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvvbw\" (UniqueName: \"kubernetes.io/projected/bd9404a7-c012-4078-8400-b4414ef2e8f1-kube-api-access-mvvbw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2\" (UID: \"bd9404a7-c012-4078-8400-b4414ef2e8f1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2"
Sep 11 17:52:46.589327 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.589288    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2aa7990-2e58-46f5-b596-1865212e0e39-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t\" (UID: \"f2aa7990-2e58-46f5-b596-1865212e0e39\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t"
Sep 11 17:52:46.589491 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.589471    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd9404a7-c012-4078-8400-b4414ef2e8f1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2\" (UID: \"bd9404a7-c012-4078-8400-b4414ef2e8f1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2"
Sep 11 17:52:46.589532 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.589496    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd9404a7-c012-4078-8400-b4414ef2e8f1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2\" (UID: \"bd9404a7-c012-4078-8400-b4414ef2e8f1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2"
Sep 11 17:52:46.597644 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.597626    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvvbw\" (UniqueName: \"kubernetes.io/projected/bd9404a7-c012-4078-8400-b4414ef2e8f1-kube-api-access-mvvbw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2\" (UID: \"bd9404a7-c012-4078-8400-b4414ef2e8f1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2"
Sep 11 17:52:46.646200 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.646169    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn"]
Sep 11 17:52:46.649863 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.649847    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn"
Sep 11 17:52:46.658149 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.658126    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn"]
Sep 11 17:52:46.664519 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.664500    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj"
Sep 11 17:52:46.690477 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.690446    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvx4v\" (UniqueName: \"kubernetes.io/projected/f2aa7990-2e58-46f5-b596-1865212e0e39-kube-api-access-mvx4v\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t\" (UID: \"f2aa7990-2e58-46f5-b596-1865212e0e39\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t"
Sep 11 17:52:46.690620 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.690492    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/098e4b2d-5127-4456-855f-d958c4d4c4c7-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn\" (UID: \"098e4b2d-5127-4456-855f-d958c4d4c4c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn"
Sep 11 17:52:46.690620 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.690516    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2aa7990-2e58-46f5-b596-1865212e0e39-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t\" (UID: \"f2aa7990-2e58-46f5-b596-1865212e0e39\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t"
Sep 11 17:52:46.690725 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.690629    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/098e4b2d-5127-4456-855f-d958c4d4c4c7-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn\" (UID: \"098e4b2d-5127-4456-855f-d958c4d4c4c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn"
Sep 11 17:52:46.690856 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.690837    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2aa7990-2e58-46f5-b596-1865212e0e39-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t\" (UID: \"f2aa7990-2e58-46f5-b596-1865212e0e39\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t"
Sep 11 17:52:46.690914 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.690897    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8fwn\" (UniqueName: \"kubernetes.io/projected/098e4b2d-5127-4456-855f-d958c4d4c4c7-kube-api-access-k8fwn\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn\" (UID: \"098e4b2d-5127-4456-855f-d958c4d4c4c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn"
Sep 11 17:52:46.691069 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.691045    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2aa7990-2e58-46f5-b596-1865212e0e39-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t\" (UID: \"f2aa7990-2e58-46f5-b596-1865212e0e39\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t"
Sep 11 17:52:46.691138 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.691118    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2aa7990-2e58-46f5-b596-1865212e0e39-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t\" (UID: \"f2aa7990-2e58-46f5-b596-1865212e0e39\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t"
Sep 11 17:52:46.699529 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.699497    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvx4v\" (UniqueName: \"kubernetes.io/projected/f2aa7990-2e58-46f5-b596-1865212e0e39-kube-api-access-mvx4v\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t\" (UID: \"f2aa7990-2e58-46f5-b596-1865212e0e39\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t"
Sep 11 17:52:46.766701 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.766665    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2"
Sep 11 17:52:46.792209 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.792180    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/098e4b2d-5127-4456-855f-d958c4d4c4c7-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn\" (UID: \"098e4b2d-5127-4456-855f-d958c4d4c4c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn"
Sep 11 17:52:46.792409 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.792216    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/098e4b2d-5127-4456-855f-d958c4d4c4c7-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn\" (UID: \"098e4b2d-5127-4456-855f-d958c4d4c4c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn"
Sep 11 17:52:46.792409 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.792309    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8fwn\" (UniqueName: \"kubernetes.io/projected/098e4b2d-5127-4456-855f-d958c4d4c4c7-kube-api-access-k8fwn\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn\" (UID: \"098e4b2d-5127-4456-855f-d958c4d4c4c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn"
Sep 11 17:52:46.792695 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.792673    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/098e4b2d-5127-4456-855f-d958c4d4c4c7-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn\" (UID: \"098e4b2d-5127-4456-855f-d958c4d4c4c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn"
Sep 11 17:52:46.792781 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.792710    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/098e4b2d-5127-4456-855f-d958c4d4c4c7-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn\" (UID: \"098e4b2d-5127-4456-855f-d958c4d4c4c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn"
Sep 11 17:52:46.801233 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.801206    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8fwn\" (UniqueName: \"kubernetes.io/projected/098e4b2d-5127-4456-855f-d958c4d4c4c7-kube-api-access-k8fwn\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn\" (UID: \"098e4b2d-5127-4456-855f-d958c4d4c4c7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn"
Sep 11 17:52:46.859674 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.859633    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t"
Sep 11 17:52:46.942719 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.942691    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj"]
Sep 11 17:52:46.944126 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:52:46.944078    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf22167a_8633_431b_a107_5f2febb8c17b.slice/crio-6bb308a01ae306302c3774bca500aa89540bd80ac76d2180a27e8894dcbee042 WatchSource:0}: Error finding container 6bb308a01ae306302c3774bca500aa89540bd80ac76d2180a27e8894dcbee042: Status 404 returned error can't find the container with id 6bb308a01ae306302c3774bca500aa89540bd80ac76d2180a27e8894dcbee042
Sep 11 17:52:46.960330 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:46.960302    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn"
Sep 11 17:52:47.045142 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:47.045119    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2"]
Sep 11 17:52:47.047883 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:52:47.047855    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9404a7_c012_4078_8400_b4414ef2e8f1.slice/crio-7f6b3b3bd174f9e73ae892806e8306171116d56a401497618aaf59ce481aced2 WatchSource:0}: Error finding container 7f6b3b3bd174f9e73ae892806e8306171116d56a401497618aaf59ce481aced2: Status 404 returned error can't find the container with id 7f6b3b3bd174f9e73ae892806e8306171116d56a401497618aaf59ce481aced2
Sep 11 17:52:47.136647 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:47.136622    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t"]
Sep 11 17:52:47.137709 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:52:47.137685    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2aa7990_2e58_46f5_b596_1865212e0e39.slice/crio-b5f5ce7b2f36faa34f58182af7f4a8f13ccdb4de4c75b62f76a75aa14c0c5336 WatchSource:0}: Error finding container b5f5ce7b2f36faa34f58182af7f4a8f13ccdb4de4c75b62f76a75aa14c0c5336: Status 404 returned error can't find the container with id b5f5ce7b2f36faa34f58182af7f4a8f13ccdb4de4c75b62f76a75aa14c0c5336
Sep 11 17:52:47.197871 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:47.197840    2559 generic.go:356] "Generic (PLEG): container finished" podID="cf22167a-8633-431b-a107-5f2febb8c17b" containerID="4c0bc30502e7360faf414cf759446a6147f78b28409c2c6b3397a46ce83b995b" exitCode=0
Sep 11 17:52:47.198003 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:47.197917    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj" event={"ID":"cf22167a-8633-431b-a107-5f2febb8c17b","Type":"ContainerDied","Data":"4c0bc30502e7360faf414cf759446a6147f78b28409c2c6b3397a46ce83b995b"}
Sep 11 17:52:47.198003 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:47.197969    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj" event={"ID":"cf22167a-8633-431b-a107-5f2febb8c17b","Type":"ContainerStarted","Data":"6bb308a01ae306302c3774bca500aa89540bd80ac76d2180a27e8894dcbee042"}
Sep 11 17:52:47.199259 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:47.199231    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t" event={"ID":"f2aa7990-2e58-46f5-b596-1865212e0e39","Type":"ContainerStarted","Data":"b5f5ce7b2f36faa34f58182af7f4a8f13ccdb4de4c75b62f76a75aa14c0c5336"}
Sep 11 17:52:47.200779 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:47.200758    2559 generic.go:356] "Generic (PLEG): container finished" podID="bd9404a7-c012-4078-8400-b4414ef2e8f1" containerID="f8669fd0d30206f09041979c17bbfd4acf982dcffde37cf3562c317f0aa5007d" exitCode=0
Sep 11 17:52:47.200897 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:47.200813    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2" event={"ID":"bd9404a7-c012-4078-8400-b4414ef2e8f1","Type":"ContainerDied","Data":"f8669fd0d30206f09041979c17bbfd4acf982dcffde37cf3562c317f0aa5007d"}
Sep 11 17:52:47.200897 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:47.200860    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2" event={"ID":"bd9404a7-c012-4078-8400-b4414ef2e8f1","Type":"ContainerStarted","Data":"7f6b3b3bd174f9e73ae892806e8306171116d56a401497618aaf59ce481aced2"}
Sep 11 17:52:47.244801 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:47.244772    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn"]
Sep 11 17:52:47.247839 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:52:47.247816    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098e4b2d_5127_4456_855f_d958c4d4c4c7.slice/crio-159f4ccd1ba63448907e6ef6bb2bc43853cecab339f79486838bc9bd4539e719 WatchSource:0}: Error finding container 159f4ccd1ba63448907e6ef6bb2bc43853cecab339f79486838bc9bd4539e719: Status 404 returned error can't find the container with id 159f4ccd1ba63448907e6ef6bb2bc43853cecab339f79486838bc9bd4539e719
Sep 11 17:52:48.207322 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:48.207289    2559 generic.go:356] "Generic (PLEG): container finished" podID="f2aa7990-2e58-46f5-b596-1865212e0e39" containerID="16e2bb0fe0c68b8933f90df4feb69b21aea4a89369d8d2bb6c261cc7fdbb6822" exitCode=0
Sep 11 17:52:48.207700 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:48.207383    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t" event={"ID":"f2aa7990-2e58-46f5-b596-1865212e0e39","Type":"ContainerDied","Data":"16e2bb0fe0c68b8933f90df4feb69b21aea4a89369d8d2bb6c261cc7fdbb6822"}
Sep 11 17:52:48.209160 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:48.209136    2559 generic.go:356] "Generic (PLEG): container finished" podID="098e4b2d-5127-4456-855f-d958c4d4c4c7" containerID="13188ba8835bfe21784b2b4fdcae2923df0eb824396bd4905636832e3e40169d" exitCode=0
Sep 11 17:52:48.209264 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:48.209218    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn" event={"ID":"098e4b2d-5127-4456-855f-d958c4d4c4c7","Type":"ContainerDied","Data":"13188ba8835bfe21784b2b4fdcae2923df0eb824396bd4905636832e3e40169d"}
Sep 11 17:52:48.209264 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:48.209258    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn" event={"ID":"098e4b2d-5127-4456-855f-d958c4d4c4c7","Type":"ContainerStarted","Data":"159f4ccd1ba63448907e6ef6bb2bc43853cecab339f79486838bc9bd4539e719"}
Sep 11 17:52:49.217100 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:49.217043    2559 generic.go:356] "Generic (PLEG): container finished" podID="bd9404a7-c012-4078-8400-b4414ef2e8f1" containerID="96586edcabc98211ee9fd87146647ad4176a619f043e2d4523575dc0bd2a38eb" exitCode=0
Sep 11 17:52:49.217479 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:49.217118    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2" event={"ID":"bd9404a7-c012-4078-8400-b4414ef2e8f1","Type":"ContainerDied","Data":"96586edcabc98211ee9fd87146647ad4176a619f043e2d4523575dc0bd2a38eb"}
Sep 11 17:52:49.219205 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:49.219146    2559 generic.go:356] "Generic (PLEG): container finished" podID="cf22167a-8633-431b-a107-5f2febb8c17b" containerID="5d7273f5e8f30e84a827f388c0413b9bcb6732bcd624aa5d18607fb1ee76c251" exitCode=0
Sep 11 17:52:49.219396 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:49.219199    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj" event={"ID":"cf22167a-8633-431b-a107-5f2febb8c17b","Type":"ContainerDied","Data":"5d7273f5e8f30e84a827f388c0413b9bcb6732bcd624aa5d18607fb1ee76c251"}
Sep 11 17:52:49.221416 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:49.221398    2559 generic.go:356] "Generic (PLEG): container finished" podID="f2aa7990-2e58-46f5-b596-1865212e0e39" containerID="ab64aa4421f2ee2352db5a16165fd14f8d314d1a578f93de1df593329ccb94f5" exitCode=0
Sep 11 17:52:49.221498 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:49.221423    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t" event={"ID":"f2aa7990-2e58-46f5-b596-1865212e0e39","Type":"ContainerDied","Data":"ab64aa4421f2ee2352db5a16165fd14f8d314d1a578f93de1df593329ccb94f5"}
Sep 11 17:52:50.229185 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:50.229150    2559 generic.go:356] "Generic (PLEG): container finished" podID="cf22167a-8633-431b-a107-5f2febb8c17b" containerID="b9ed930441c405467004292a4a34819553490263ea9a1b4885595605d08cad6f" exitCode=0
Sep 11 17:52:50.229659 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:50.229241    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj" event={"ID":"cf22167a-8633-431b-a107-5f2febb8c17b","Type":"ContainerDied","Data":"b9ed930441c405467004292a4a34819553490263ea9a1b4885595605d08cad6f"}
Sep 11 17:52:50.231258 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:50.231240    2559 generic.go:356] "Generic (PLEG): container finished" podID="f2aa7990-2e58-46f5-b596-1865212e0e39" containerID="e9bff9343ad5a9816a0a106d43543b4bc698ac712ffdce282adf4df47eedceff" exitCode=0
Sep 11 17:52:50.231335 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:50.231270    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t" event={"ID":"f2aa7990-2e58-46f5-b596-1865212e0e39","Type":"ContainerDied","Data":"e9bff9343ad5a9816a0a106d43543b4bc698ac712ffdce282adf4df47eedceff"}
Sep 11 17:52:50.233215 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:50.233191    2559 generic.go:356] "Generic (PLEG): container finished" podID="098e4b2d-5127-4456-855f-d958c4d4c4c7" containerID="c4039d3a744b27a18ce626fff4a0d6823eff04bd551d23204b1a469156a335c2" exitCode=0
Sep 11 17:52:50.233294 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:50.233256    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn" event={"ID":"098e4b2d-5127-4456-855f-d958c4d4c4c7","Type":"ContainerDied","Data":"c4039d3a744b27a18ce626fff4a0d6823eff04bd551d23204b1a469156a335c2"}
Sep 11 17:52:50.235546 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:50.235527    2559 generic.go:356] "Generic (PLEG): container finished" podID="bd9404a7-c012-4078-8400-b4414ef2e8f1" containerID="cfb46649be69e204d890cbc655cbb1771b31c707c8b7972a4f836957362f0ac0" exitCode=0
Sep 11 17:52:50.235629 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:50.235607    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2" event={"ID":"bd9404a7-c012-4078-8400-b4414ef2e8f1","Type":"ContainerDied","Data":"cfb46649be69e204d890cbc655cbb1771b31c707c8b7972a4f836957362f0ac0"}
Sep 11 17:52:51.242132 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.242094    2559 generic.go:356] "Generic (PLEG): container finished" podID="098e4b2d-5127-4456-855f-d958c4d4c4c7" containerID="7ce30af274a9b78d5e55284a3069ef12dad99cbed14f41cbc6f636adaaa6a774" exitCode=0
Sep 11 17:52:51.242533 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.242161    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn" event={"ID":"098e4b2d-5127-4456-855f-d958c4d4c4c7","Type":"ContainerDied","Data":"7ce30af274a9b78d5e55284a3069ef12dad99cbed14f41cbc6f636adaaa6a774"}
Sep 11 17:52:51.367649 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.367622    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t"
Sep 11 17:52:51.414265 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.414236    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj"
Sep 11 17:52:51.433312 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.433286    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf22167a-8633-431b-a107-5f2febb8c17b-bundle\") pod \"cf22167a-8633-431b-a107-5f2febb8c17b\" (UID: \"cf22167a-8633-431b-a107-5f2febb8c17b\") "
Sep 11 17:52:51.433433 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.433323    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvx4v\" (UniqueName: \"kubernetes.io/projected/f2aa7990-2e58-46f5-b596-1865212e0e39-kube-api-access-mvx4v\") pod \"f2aa7990-2e58-46f5-b596-1865212e0e39\" (UID: \"f2aa7990-2e58-46f5-b596-1865212e0e39\") "
Sep 11 17:52:51.433433 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.433372    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf22167a-8633-431b-a107-5f2febb8c17b-util\") pod \"cf22167a-8633-431b-a107-5f2febb8c17b\" (UID: \"cf22167a-8633-431b-a107-5f2febb8c17b\") "
Sep 11 17:52:51.433522 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.433437    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2aa7990-2e58-46f5-b596-1865212e0e39-bundle\") pod \"f2aa7990-2e58-46f5-b596-1865212e0e39\" (UID: \"f2aa7990-2e58-46f5-b596-1865212e0e39\") "
Sep 11 17:52:51.433522 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.433472    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2aa7990-2e58-46f5-b596-1865212e0e39-util\") pod \"f2aa7990-2e58-46f5-b596-1865212e0e39\" (UID: \"f2aa7990-2e58-46f5-b596-1865212e0e39\") "
Sep 11 17:52:51.433522 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.433510    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbn7r\" (UniqueName: \"kubernetes.io/projected/cf22167a-8633-431b-a107-5f2febb8c17b-kube-api-access-dbn7r\") pod \"cf22167a-8633-431b-a107-5f2febb8c17b\" (UID: \"cf22167a-8633-431b-a107-5f2febb8c17b\") "
Sep 11 17:52:51.434281 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.434105    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf22167a-8633-431b-a107-5f2febb8c17b-bundle" (OuterVolumeSpecName: "bundle") pod "cf22167a-8633-431b-a107-5f2febb8c17b" (UID: "cf22167a-8633-431b-a107-5f2febb8c17b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:52:51.434281 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.434198    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2aa7990-2e58-46f5-b596-1865212e0e39-bundle" (OuterVolumeSpecName: "bundle") pod "f2aa7990-2e58-46f5-b596-1865212e0e39" (UID: "f2aa7990-2e58-46f5-b596-1865212e0e39"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:52:51.436916 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.436657    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2aa7990-2e58-46f5-b596-1865212e0e39-kube-api-access-mvx4v" (OuterVolumeSpecName: "kube-api-access-mvx4v") pod "f2aa7990-2e58-46f5-b596-1865212e0e39" (UID: "f2aa7990-2e58-46f5-b596-1865212e0e39"). InnerVolumeSpecName "kube-api-access-mvx4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:52:51.437769 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.437723    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf22167a-8633-431b-a107-5f2febb8c17b-kube-api-access-dbn7r" (OuterVolumeSpecName: "kube-api-access-dbn7r") pod "cf22167a-8633-431b-a107-5f2febb8c17b" (UID: "cf22167a-8633-431b-a107-5f2febb8c17b"). InnerVolumeSpecName "kube-api-access-dbn7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:52:51.440628 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.440597    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2aa7990-2e58-46f5-b596-1865212e0e39-util" (OuterVolumeSpecName: "util") pod "f2aa7990-2e58-46f5-b596-1865212e0e39" (UID: "f2aa7990-2e58-46f5-b596-1865212e0e39"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:52:51.441308 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.441285    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf22167a-8633-431b-a107-5f2febb8c17b-util" (OuterVolumeSpecName: "util") pod "cf22167a-8633-431b-a107-5f2febb8c17b" (UID: "cf22167a-8633-431b-a107-5f2febb8c17b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:52:51.442136 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.442121    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2"
Sep 11 17:52:51.534946 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.534859    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvvbw\" (UniqueName: \"kubernetes.io/projected/bd9404a7-c012-4078-8400-b4414ef2e8f1-kube-api-access-mvvbw\") pod \"bd9404a7-c012-4078-8400-b4414ef2e8f1\" (UID: \"bd9404a7-c012-4078-8400-b4414ef2e8f1\") "
Sep 11 17:52:51.534946 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.534938    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd9404a7-c012-4078-8400-b4414ef2e8f1-bundle\") pod \"bd9404a7-c012-4078-8400-b4414ef2e8f1\" (UID: \"bd9404a7-c012-4078-8400-b4414ef2e8f1\") "
Sep 11 17:52:51.535140 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.535046    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd9404a7-c012-4078-8400-b4414ef2e8f1-util\") pod \"bd9404a7-c012-4078-8400-b4414ef2e8f1\" (UID: \"bd9404a7-c012-4078-8400-b4414ef2e8f1\") "
Sep 11 17:52:51.535408 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.535387    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbn7r\" (UniqueName: \"kubernetes.io/projected/cf22167a-8633-431b-a107-5f2febb8c17b-kube-api-access-dbn7r\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:52:51.535408 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.535408    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf22167a-8633-431b-a107-5f2febb8c17b-bundle\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:52:51.535545 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.535418    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvx4v\" (UniqueName: \"kubernetes.io/projected/f2aa7990-2e58-46f5-b596-1865212e0e39-kube-api-access-mvx4v\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:52:51.535545 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.535428    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf22167a-8633-431b-a107-5f2febb8c17b-util\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:52:51.535545 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.535437    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2aa7990-2e58-46f5-b596-1865212e0e39-bundle\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:52:51.535545 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.535444    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2aa7990-2e58-46f5-b596-1865212e0e39-util\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:52:51.535733 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.535582    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9404a7-c012-4078-8400-b4414ef2e8f1-bundle" (OuterVolumeSpecName: "bundle") pod "bd9404a7-c012-4078-8400-b4414ef2e8f1" (UID: "bd9404a7-c012-4078-8400-b4414ef2e8f1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:52:51.537097 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.537074    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9404a7-c012-4078-8400-b4414ef2e8f1-kube-api-access-mvvbw" (OuterVolumeSpecName: "kube-api-access-mvvbw") pod "bd9404a7-c012-4078-8400-b4414ef2e8f1" (UID: "bd9404a7-c012-4078-8400-b4414ef2e8f1"). InnerVolumeSpecName "kube-api-access-mvvbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:52:51.540114 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.540092    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9404a7-c012-4078-8400-b4414ef2e8f1-util" (OuterVolumeSpecName: "util") pod "bd9404a7-c012-4078-8400-b4414ef2e8f1" (UID: "bd9404a7-c012-4078-8400-b4414ef2e8f1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:52:51.636873 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.636834    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvvbw\" (UniqueName: \"kubernetes.io/projected/bd9404a7-c012-4078-8400-b4414ef2e8f1-kube-api-access-mvvbw\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:52:51.636873 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.636867    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd9404a7-c012-4078-8400-b4414ef2e8f1-bundle\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:52:51.636873 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:51.636878    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd9404a7-c012-4078-8400-b4414ef2e8f1-util\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:52:52.249259 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:52.249230    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2"
Sep 11 17:52:52.249259 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:52.249244    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9bwr2" event={"ID":"bd9404a7-c012-4078-8400-b4414ef2e8f1","Type":"ContainerDied","Data":"7f6b3b3bd174f9e73ae892806e8306171116d56a401497618aaf59ce481aced2"}
Sep 11 17:52:52.249769 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:52.249292    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f6b3b3bd174f9e73ae892806e8306171116d56a401497618aaf59ce481aced2"
Sep 11 17:52:52.251384 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:52.251359    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj"
Sep 11 17:52:52.251497 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:52.251383    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xfrxj" event={"ID":"cf22167a-8633-431b-a107-5f2febb8c17b","Type":"ContainerDied","Data":"6bb308a01ae306302c3774bca500aa89540bd80ac76d2180a27e8894dcbee042"}
Sep 11 17:52:52.251497 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:52.251430    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bb308a01ae306302c3774bca500aa89540bd80ac76d2180a27e8894dcbee042"
Sep 11 17:52:52.253675 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:52.253655    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t"
Sep 11 17:52:52.253770 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:52.253669    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmf44t" event={"ID":"f2aa7990-2e58-46f5-b596-1865212e0e39","Type":"ContainerDied","Data":"b5f5ce7b2f36faa34f58182af7f4a8f13ccdb4de4c75b62f76a75aa14c0c5336"}
Sep 11 17:52:52.253770 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:52.253714    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5f5ce7b2f36faa34f58182af7f4a8f13ccdb4de4c75b62f76a75aa14c0c5336"
Sep 11 17:52:52.386413 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:52.386392    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn"
Sep 11 17:52:52.443219 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:52.443190    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/098e4b2d-5127-4456-855f-d958c4d4c4c7-bundle\") pod \"098e4b2d-5127-4456-855f-d958c4d4c4c7\" (UID: \"098e4b2d-5127-4456-855f-d958c4d4c4c7\") "
Sep 11 17:52:52.443443 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:52.443252    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/098e4b2d-5127-4456-855f-d958c4d4c4c7-util\") pod \"098e4b2d-5127-4456-855f-d958c4d4c4c7\" (UID: \"098e4b2d-5127-4456-855f-d958c4d4c4c7\") "
Sep 11 17:52:52.443443 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:52.443281    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8fwn\" (UniqueName: \"kubernetes.io/projected/098e4b2d-5127-4456-855f-d958c4d4c4c7-kube-api-access-k8fwn\") pod \"098e4b2d-5127-4456-855f-d958c4d4c4c7\" (UID: \"098e4b2d-5127-4456-855f-d958c4d4c4c7\") "
Sep 11 17:52:52.443717 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:52.443694    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098e4b2d-5127-4456-855f-d958c4d4c4c7-bundle" (OuterVolumeSpecName: "bundle") pod "098e4b2d-5127-4456-855f-d958c4d4c4c7" (UID: "098e4b2d-5127-4456-855f-d958c4d4c4c7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:52:52.445430 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:52.445407    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098e4b2d-5127-4456-855f-d958c4d4c4c7-kube-api-access-k8fwn" (OuterVolumeSpecName: "kube-api-access-k8fwn") pod "098e4b2d-5127-4456-855f-d958c4d4c4c7" (UID: "098e4b2d-5127-4456-855f-d958c4d4c4c7"). InnerVolumeSpecName "kube-api-access-k8fwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:52:52.448559 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:52.448537    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098e4b2d-5127-4456-855f-d958c4d4c4c7-util" (OuterVolumeSpecName: "util") pod "098e4b2d-5127-4456-855f-d958c4d4c4c7" (UID: "098e4b2d-5127-4456-855f-d958c4d4c4c7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:52:52.544101 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:52.544014    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/098e4b2d-5127-4456-855f-d958c4d4c4c7-util\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:52:52.544101 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:52.544044    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8fwn\" (UniqueName: \"kubernetes.io/projected/098e4b2d-5127-4456-855f-d958c4d4c4c7-kube-api-access-k8fwn\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:52:52.544101 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:52.544058    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/098e4b2d-5127-4456-855f-d958c4d4c4c7-bundle\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:52:53.261011 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:53.260979    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn"
Sep 11 17:52:53.261011 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:53.260998    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g27tn" event={"ID":"098e4b2d-5127-4456-855f-d958c4d4c4c7","Type":"ContainerDied","Data":"159f4ccd1ba63448907e6ef6bb2bc43853cecab339f79486838bc9bd4539e719"}
Sep 11 17:52:53.261536 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:52:53.261049    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="159f4ccd1ba63448907e6ef6bb2bc43853cecab339f79486838bc9bd4539e719"
Sep 11 17:53:12.609796 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.609757    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-7jtn6"]
Sep 11 17:53:12.610211 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610078    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf22167a-8633-431b-a107-5f2febb8c17b" containerName="util"
Sep 11 17:53:12.610211 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610088    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf22167a-8633-431b-a107-5f2febb8c17b" containerName="util"
Sep 11 17:53:12.610211 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610098    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="098e4b2d-5127-4456-855f-d958c4d4c4c7" containerName="extract"
Sep 11 17:53:12.610211 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610103    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="098e4b2d-5127-4456-855f-d958c4d4c4c7" containerName="extract"
Sep 11 17:53:12.610211 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610112    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf22167a-8633-431b-a107-5f2febb8c17b" containerName="extract"
Sep 11 17:53:12.610211 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610118    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf22167a-8633-431b-a107-5f2febb8c17b" containerName="extract"
Sep 11 17:53:12.610211 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610126    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd9404a7-c012-4078-8400-b4414ef2e8f1" containerName="util"
Sep 11 17:53:12.610211 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610131    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd9404a7-c012-4078-8400-b4414ef2e8f1" containerName="util"
Sep 11 17:53:12.610211 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610139    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd9404a7-c012-4078-8400-b4414ef2e8f1" containerName="extract"
Sep 11 17:53:12.610211 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610146    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd9404a7-c012-4078-8400-b4414ef2e8f1" containerName="extract"
Sep 11 17:53:12.610211 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610154    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf22167a-8633-431b-a107-5f2febb8c17b" containerName="pull"
Sep 11 17:53:12.610211 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610158    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf22167a-8633-431b-a107-5f2febb8c17b" containerName="pull"
Sep 11 17:53:12.610211 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610166    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd9404a7-c012-4078-8400-b4414ef2e8f1" containerName="pull"
Sep 11 17:53:12.610211 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610171    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd9404a7-c012-4078-8400-b4414ef2e8f1" containerName="pull"
Sep 11 17:53:12.610211 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610181    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="098e4b2d-5127-4456-855f-d958c4d4c4c7" containerName="util"
Sep 11 17:53:12.610211 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610186    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="098e4b2d-5127-4456-855f-d958c4d4c4c7" containerName="util"
Sep 11 17:53:12.610211 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610192    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2aa7990-2e58-46f5-b596-1865212e0e39" containerName="util"
Sep 11 17:53:12.610211 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610197    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2aa7990-2e58-46f5-b596-1865212e0e39" containerName="util"
Sep 11 17:53:12.610211 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610205    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2aa7990-2e58-46f5-b596-1865212e0e39" containerName="extract"
Sep 11 17:53:12.610211 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610210    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2aa7990-2e58-46f5-b596-1865212e0e39" containerName="extract"
Sep 11 17:53:12.610211 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610216    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="098e4b2d-5127-4456-855f-d958c4d4c4c7" containerName="pull"
Sep 11 17:53:12.610211 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610221    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="098e4b2d-5127-4456-855f-d958c4d4c4c7" containerName="pull"
Sep 11 17:53:12.610864 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610229    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2aa7990-2e58-46f5-b596-1865212e0e39" containerName="pull"
Sep 11 17:53:12.610864 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610234    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2aa7990-2e58-46f5-b596-1865212e0e39" containerName="pull"
Sep 11 17:53:12.610864 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610280    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2aa7990-2e58-46f5-b596-1865212e0e39" containerName="extract"
Sep 11 17:53:12.610864 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610289    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd9404a7-c012-4078-8400-b4414ef2e8f1" containerName="extract"
Sep 11 17:53:12.610864 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610296    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf22167a-8633-431b-a107-5f2febb8c17b" containerName="extract"
Sep 11 17:53:12.610864 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.610302    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="098e4b2d-5127-4456-855f-d958c4d4c4c7" containerName="extract"
Sep 11 17:53:12.614588 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.614564    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7jtn6"
Sep 11 17:53:12.618484 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.618460    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 11 17:53:12.618996 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.618981    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 17:53:12.619162 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.619147    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 11 17:53:12.619693 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.619671    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 17:53:12.626848 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.626831    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-667qn\""
Sep 11 17:53:12.643574 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.643545    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-7jtn6"]
Sep 11 17:53:12.725983 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.725933    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c88ww\" (UniqueName: \"kubernetes.io/projected/28fa71d6-fe51-46e4-8664-02f82843e893-kube-api-access-c88ww\") pod \"kuadrant-console-plugin-fbdd4f445-7jtn6\" (UID: \"28fa71d6-fe51-46e4-8664-02f82843e893\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7jtn6"
Sep 11 17:53:12.725983 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.725984    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/28fa71d6-fe51-46e4-8664-02f82843e893-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-7jtn6\" (UID: \"28fa71d6-fe51-46e4-8664-02f82843e893\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7jtn6"
Sep 11 17:53:12.726195 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.726013    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/28fa71d6-fe51-46e4-8664-02f82843e893-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-7jtn6\" (UID: \"28fa71d6-fe51-46e4-8664-02f82843e893\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7jtn6"
Sep 11 17:53:12.827468 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.827430    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c88ww\" (UniqueName: \"kubernetes.io/projected/28fa71d6-fe51-46e4-8664-02f82843e893-kube-api-access-c88ww\") pod \"kuadrant-console-plugin-fbdd4f445-7jtn6\" (UID: \"28fa71d6-fe51-46e4-8664-02f82843e893\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7jtn6"
Sep 11 17:53:12.827652 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.827478    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/28fa71d6-fe51-46e4-8664-02f82843e893-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-7jtn6\" (UID: \"28fa71d6-fe51-46e4-8664-02f82843e893\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7jtn6"
Sep 11 17:53:12.827652 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.827511    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/28fa71d6-fe51-46e4-8664-02f82843e893-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-7jtn6\" (UID: \"28fa71d6-fe51-46e4-8664-02f82843e893\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7jtn6"
Sep 11 17:53:12.828143 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.828113    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/28fa71d6-fe51-46e4-8664-02f82843e893-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-7jtn6\" (UID: \"28fa71d6-fe51-46e4-8664-02f82843e893\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7jtn6"
Sep 11 17:53:12.830128 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.830101    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/28fa71d6-fe51-46e4-8664-02f82843e893-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-7jtn6\" (UID: \"28fa71d6-fe51-46e4-8664-02f82843e893\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7jtn6"
Sep 11 17:53:12.839606 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.839578    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c88ww\" (UniqueName: \"kubernetes.io/projected/28fa71d6-fe51-46e4-8664-02f82843e893-kube-api-access-c88ww\") pod \"kuadrant-console-plugin-fbdd4f445-7jtn6\" (UID: \"28fa71d6-fe51-46e4-8664-02f82843e893\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7jtn6"
Sep 11 17:53:12.923677 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:12.923642    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7jtn6"
Sep 11 17:53:13.242716 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:13.242625    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-7jtn6"]
Sep 11 17:53:13.246724 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:53:13.246686    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28fa71d6_fe51_46e4_8664_02f82843e893.slice/crio-3585383bad618ddb5578ea9a9791f0d18ce480b05bbfffbd4d8a38504678ce02 WatchSource:0}: Error finding container 3585383bad618ddb5578ea9a9791f0d18ce480b05bbfffbd4d8a38504678ce02: Status 404 returned error can't find the container with id 3585383bad618ddb5578ea9a9791f0d18ce480b05bbfffbd4d8a38504678ce02
Sep 11 17:53:13.360003 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:13.359971    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7jtn6" event={"ID":"28fa71d6-fe51-46e4-8664-02f82843e893","Type":"ContainerStarted","Data":"3585383bad618ddb5578ea9a9791f0d18ce480b05bbfffbd4d8a38504678ce02"}
Sep 11 17:53:18.388964 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:18.388919    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7jtn6" event={"ID":"28fa71d6-fe51-46e4-8664-02f82843e893","Type":"ContainerStarted","Data":"29354ac63fffa10dc36ab689f67405bb63679772091d1cc7ca6c51d081219529"}
Sep 11 17:53:18.442786 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:53:18.442722    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7jtn6" podStartSLOduration=1.442817922 podStartE2EDuration="6.442707518s" totalImagesPullingTime="4.999889596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:53:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:53:18.440494749 +0000 UTC m=+503.885595320" watchObservedRunningTime="2026-09-11 17:53:18.442707518 +0000 UTC m=+503.887808088"
Sep 11 17:54:01.750671 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:01.750636    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 17:54:01.754156 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:01.754137    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"
Sep 11 17:54:01.757770 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:01.757749    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 17:54:01.780851 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:01.780815    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 17:54:01.828171 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:01.828136    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 17:54:01.854705 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:01.854675    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7eaff0ad-e5ca-4029-99bc-4835f2dd95c1-config-file\") pod \"limitador-limitador-84994f7ffc-dfgnf\" (UID: \"7eaff0ad-e5ca-4029-99bc-4835f2dd95c1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"
Sep 11 17:54:01.854895 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:01.854795    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgzx7\" (UniqueName: \"kubernetes.io/projected/7eaff0ad-e5ca-4029-99bc-4835f2dd95c1-kube-api-access-hgzx7\") pod \"limitador-limitador-84994f7ffc-dfgnf\" (UID: \"7eaff0ad-e5ca-4029-99bc-4835f2dd95c1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"
Sep 11 17:54:01.955735 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:01.955702    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7eaff0ad-e5ca-4029-99bc-4835f2dd95c1-config-file\") pod \"limitador-limitador-84994f7ffc-dfgnf\" (UID: \"7eaff0ad-e5ca-4029-99bc-4835f2dd95c1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"
Sep 11 17:54:01.955950 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:01.955780    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgzx7\" (UniqueName: \"kubernetes.io/projected/7eaff0ad-e5ca-4029-99bc-4835f2dd95c1-kube-api-access-hgzx7\") pod \"limitador-limitador-84994f7ffc-dfgnf\" (UID: \"7eaff0ad-e5ca-4029-99bc-4835f2dd95c1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"
Sep 11 17:54:01.956540 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:01.956514    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7eaff0ad-e5ca-4029-99bc-4835f2dd95c1-config-file\") pod \"limitador-limitador-84994f7ffc-dfgnf\" (UID: \"7eaff0ad-e5ca-4029-99bc-4835f2dd95c1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"
Sep 11 17:54:01.974026 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:01.973993    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgzx7\" (UniqueName: \"kubernetes.io/projected/7eaff0ad-e5ca-4029-99bc-4835f2dd95c1-kube-api-access-hgzx7\") pod \"limitador-limitador-84994f7ffc-dfgnf\" (UID: \"7eaff0ad-e5ca-4029-99bc-4835f2dd95c1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"
Sep 11 17:54:02.063650 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:02.063551    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"
Sep 11 17:54:02.338817 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:02.337829    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dn9mg"]
Sep 11 17:54:02.342776 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:02.342746    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-dn9mg"
Sep 11 17:54:02.346198 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:02.346171    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5z7pt\""
Sep 11 17:54:02.354560 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:54:02.354536    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eaff0ad_e5ca_4029_99bc_4835f2dd95c1.slice/crio-8c64e25b196dd462aaeff949ff1cf5c926d3df925208e993ddcc91ba493dd2ec WatchSource:0}: Error finding container 8c64e25b196dd462aaeff949ff1cf5c926d3df925208e993ddcc91ba493dd2ec: Status 404 returned error can't find the container with id 8c64e25b196dd462aaeff949ff1cf5c926d3df925208e993ddcc91ba493dd2ec
Sep 11 17:54:02.355056 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:02.355031    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 17:54:02.357994 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:02.357966    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dn9mg"]
Sep 11 17:54:02.460097 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:02.460058    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfwrp\" (UniqueName: \"kubernetes.io/projected/ccee66ec-f70e-490a-9aba-718f49e6e4e2-kube-api-access-pfwrp\") pod \"authorino-7bcfd8f9cd-dn9mg\" (UID: \"ccee66ec-f70e-490a-9aba-718f49e6e4e2\") " pod="kuadrant-system/authorino-7bcfd8f9cd-dn9mg"
Sep 11 17:54:02.560998 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:02.560958    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfwrp\" (UniqueName: \"kubernetes.io/projected/ccee66ec-f70e-490a-9aba-718f49e6e4e2-kube-api-access-pfwrp\") pod \"authorino-7bcfd8f9cd-dn9mg\" (UID: \"ccee66ec-f70e-490a-9aba-718f49e6e4e2\") " pod="kuadrant-system/authorino-7bcfd8f9cd-dn9mg"
Sep 11 17:54:02.577128 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:02.577099    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfwrp\" (UniqueName: \"kubernetes.io/projected/ccee66ec-f70e-490a-9aba-718f49e6e4e2-kube-api-access-pfwrp\") pod \"authorino-7bcfd8f9cd-dn9mg\" (UID: \"ccee66ec-f70e-490a-9aba-718f49e6e4e2\") " pod="kuadrant-system/authorino-7bcfd8f9cd-dn9mg"
Sep 11 17:54:02.606627 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:02.606550    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-dfgnf" event={"ID":"7eaff0ad-e5ca-4029-99bc-4835f2dd95c1","Type":"ContainerStarted","Data":"8c64e25b196dd462aaeff949ff1cf5c926d3df925208e993ddcc91ba493dd2ec"}
Sep 11 17:54:02.617395 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:02.617369    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-7k4b4"]
Sep 11 17:54:02.620765 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:02.620748    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-7k4b4"
Sep 11 17:54:02.633885 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:02.633858    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-7k4b4"]
Sep 11 17:54:02.654590 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:02.654565    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-dn9mg"
Sep 11 17:54:02.762586 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:02.762548    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz4ml\" (UniqueName: \"kubernetes.io/projected/89668824-6889-4b63-9cd2-ea109021d98c-kube-api-access-dz4ml\") pod \"authorino-59c974bd48-7k4b4\" (UID: \"89668824-6889-4b63-9cd2-ea109021d98c\") " pod="kuadrant-system/authorino-59c974bd48-7k4b4"
Sep 11 17:54:02.864063 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:02.863987    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz4ml\" (UniqueName: \"kubernetes.io/projected/89668824-6889-4b63-9cd2-ea109021d98c-kube-api-access-dz4ml\") pod \"authorino-59c974bd48-7k4b4\" (UID: \"89668824-6889-4b63-9cd2-ea109021d98c\") " pod="kuadrant-system/authorino-59c974bd48-7k4b4"
Sep 11 17:54:02.874471 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:02.874442    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz4ml\" (UniqueName: \"kubernetes.io/projected/89668824-6889-4b63-9cd2-ea109021d98c-kube-api-access-dz4ml\") pod \"authorino-59c974bd48-7k4b4\" (UID: \"89668824-6889-4b63-9cd2-ea109021d98c\") " pod="kuadrant-system/authorino-59c974bd48-7k4b4"
Sep 11 17:54:02.930102 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:02.930068    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-7k4b4"
Sep 11 17:54:02.930237 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:02.930115    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dn9mg"]
Sep 11 17:54:02.931116 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:54:02.931091    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccee66ec_f70e_490a_9aba_718f49e6e4e2.slice/crio-3beb7d977edfeb58928e41bfdf3458240ca8eba3d9e956525cb3adf72af71126 WatchSource:0}: Error finding container 3beb7d977edfeb58928e41bfdf3458240ca8eba3d9e956525cb3adf72af71126: Status 404 returned error can't find the container with id 3beb7d977edfeb58928e41bfdf3458240ca8eba3d9e956525cb3adf72af71126
Sep 11 17:54:03.218581 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:03.218333    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-7k4b4"]
Sep 11 17:54:03.221088 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:54:03.221055    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89668824_6889_4b63_9cd2_ea109021d98c.slice/crio-7e6da6c839c32c613f4f2f3e1b2303383fe5344c8b0ce1e666afd9cd71a185c6 WatchSource:0}: Error finding container 7e6da6c839c32c613f4f2f3e1b2303383fe5344c8b0ce1e666afd9cd71a185c6: Status 404 returned error can't find the container with id 7e6da6c839c32c613f4f2f3e1b2303383fe5344c8b0ce1e666afd9cd71a185c6
Sep 11 17:54:03.612860 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:03.612767    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-dn9mg" event={"ID":"ccee66ec-f70e-490a-9aba-718f49e6e4e2","Type":"ContainerStarted","Data":"3beb7d977edfeb58928e41bfdf3458240ca8eba3d9e956525cb3adf72af71126"}
Sep 11 17:54:03.614123 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:03.614094    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-7k4b4" event={"ID":"89668824-6889-4b63-9cd2-ea109021d98c","Type":"ContainerStarted","Data":"7e6da6c839c32c613f4f2f3e1b2303383fe5344c8b0ce1e666afd9cd71a185c6"}
Sep 11 17:54:06.633955 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:06.633906    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-dfgnf" event={"ID":"7eaff0ad-e5ca-4029-99bc-4835f2dd95c1","Type":"ContainerStarted","Data":"4238875846fa94bd6f3fa513181cfd9c8bf1fb1a87e1e635997d2ef7639aa6b0"}
Sep 11 17:54:06.634470 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:06.634018    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"
Sep 11 17:54:06.635624 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:06.635599    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-dn9mg" event={"ID":"ccee66ec-f70e-490a-9aba-718f49e6e4e2","Type":"ContainerStarted","Data":"1e871dd997aec3bb6ecee085b366c5340ef72d486e75fe72041098e4ef51ae08"}
Sep 11 17:54:06.637078 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:06.637060    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-7k4b4" event={"ID":"89668824-6889-4b63-9cd2-ea109021d98c","Type":"ContainerStarted","Data":"bb07500fd00977d203503c03b804d3163382762070b3a1280b7b69d175389f2d"}
Sep 11 17:54:06.654232 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:06.654189    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-dfgnf" podStartSLOduration=1.961222402 podStartE2EDuration="5.654175696s" totalImagesPullingTime="3.692953294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:06.652080327 +0000 UTC m=+552.097180900" watchObservedRunningTime="2026-09-11 17:54:06.654175696 +0000 UTC m=+552.099276297"
Sep 11 17:54:06.669462 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:06.669422    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-dn9mg" podStartSLOduration=1.5483046059999999 podStartE2EDuration="4.669409731s" totalImagesPullingTime="3.121105125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:06.667257231 +0000 UTC m=+552.112357824" watchObservedRunningTime="2026-09-11 17:54:06.669409731 +0000 UTC m=+552.114510298"
Sep 11 17:54:06.682657 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:06.682597    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-7k4b4" podStartSLOduration=1.840494364 podStartE2EDuration="4.682576035s" totalImagesPullingTime="2.842081671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:06.681959816 +0000 UTC m=+552.127060387" watchObservedRunningTime="2026-09-11 17:54:06.682576035 +0000 UTC m=+552.127676606"
Sep 11 17:54:06.717413 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:06.717377    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dn9mg"]
Sep 11 17:54:08.647428 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:08.647389    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-dn9mg" podUID="ccee66ec-f70e-490a-9aba-718f49e6e4e2" containerName="authorino" containerID="cri-o://1e871dd997aec3bb6ecee085b366c5340ef72d486e75fe72041098e4ef51ae08" gracePeriod=30
Sep 11 17:54:08.898954 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:08.898892    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-dn9mg"
Sep 11 17:54:09.021930 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:09.021887    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfwrp\" (UniqueName: \"kubernetes.io/projected/ccee66ec-f70e-490a-9aba-718f49e6e4e2-kube-api-access-pfwrp\") pod \"ccee66ec-f70e-490a-9aba-718f49e6e4e2\" (UID: \"ccee66ec-f70e-490a-9aba-718f49e6e4e2\") "
Sep 11 17:54:09.024218 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:09.024192    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccee66ec-f70e-490a-9aba-718f49e6e4e2-kube-api-access-pfwrp" (OuterVolumeSpecName: "kube-api-access-pfwrp") pod "ccee66ec-f70e-490a-9aba-718f49e6e4e2" (UID: "ccee66ec-f70e-490a-9aba-718f49e6e4e2"). InnerVolumeSpecName "kube-api-access-pfwrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:54:09.123188 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:09.123146    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfwrp\" (UniqueName: \"kubernetes.io/projected/ccee66ec-f70e-490a-9aba-718f49e6e4e2-kube-api-access-pfwrp\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:54:09.653269 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:09.653232    2559 generic.go:356] "Generic (PLEG): container finished" podID="ccee66ec-f70e-490a-9aba-718f49e6e4e2" containerID="1e871dd997aec3bb6ecee085b366c5340ef72d486e75fe72041098e4ef51ae08" exitCode=0
Sep 11 17:54:09.653741 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:09.653271    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-dn9mg"
Sep 11 17:54:09.653741 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:09.653290    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-dn9mg" event={"ID":"ccee66ec-f70e-490a-9aba-718f49e6e4e2","Type":"ContainerDied","Data":"1e871dd997aec3bb6ecee085b366c5340ef72d486e75fe72041098e4ef51ae08"}
Sep 11 17:54:09.653741 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:09.653321    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-dn9mg" event={"ID":"ccee66ec-f70e-490a-9aba-718f49e6e4e2","Type":"ContainerDied","Data":"3beb7d977edfeb58928e41bfdf3458240ca8eba3d9e956525cb3adf72af71126"}
Sep 11 17:54:09.653741 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:09.653339    2559 scope.go:117] "RemoveContainer" containerID="1e871dd997aec3bb6ecee085b366c5340ef72d486e75fe72041098e4ef51ae08"
Sep 11 17:54:09.661871 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:09.661851    2559 scope.go:117] "RemoveContainer" containerID="1e871dd997aec3bb6ecee085b366c5340ef72d486e75fe72041098e4ef51ae08"
Sep 11 17:54:09.662173 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:54:09.662153    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e871dd997aec3bb6ecee085b366c5340ef72d486e75fe72041098e4ef51ae08\": container with ID starting with 1e871dd997aec3bb6ecee085b366c5340ef72d486e75fe72041098e4ef51ae08 not found: ID does not exist" containerID="1e871dd997aec3bb6ecee085b366c5340ef72d486e75fe72041098e4ef51ae08"
Sep 11 17:54:09.662239 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:09.662183    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e871dd997aec3bb6ecee085b366c5340ef72d486e75fe72041098e4ef51ae08"} err="failed to get container status \"1e871dd997aec3bb6ecee085b366c5340ef72d486e75fe72041098e4ef51ae08\": rpc error: code = NotFound desc = could not find container \"1e871dd997aec3bb6ecee085b366c5340ef72d486e75fe72041098e4ef51ae08\": container with ID starting with 1e871dd997aec3bb6ecee085b366c5340ef72d486e75fe72041098e4ef51ae08 not found: ID does not exist"
Sep 11 17:54:09.677497 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:09.677474    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dn9mg"]
Sep 11 17:54:09.685455 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:09.685429    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dn9mg"]
Sep 11 17:54:11.052585 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:11.052551    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccee66ec-f70e-490a-9aba-718f49e6e4e2" path="/var/lib/kubelet/pods/ccee66ec-f70e-490a-9aba-718f49e6e4e2/volumes"
Sep 11 17:54:17.643904 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:17.643873    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"
Sep 11 17:54:36.128310 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.128263    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-2mfqq"]
Sep 11 17:54:36.128740 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.128659    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccee66ec-f70e-490a-9aba-718f49e6e4e2" containerName="authorino"
Sep 11 17:54:36.128740 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.128673    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccee66ec-f70e-490a-9aba-718f49e6e4e2" containerName="authorino"
Sep 11 17:54:36.128816 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.128748    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccee66ec-f70e-490a-9aba-718f49e6e4e2" containerName="authorino"
Sep 11 17:54:36.131773 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.131755    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-2mfqq"
Sep 11 17:54:36.142168 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.142144    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-2mfqq"]
Sep 11 17:54:36.236819 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.236781    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-2mfqq"]
Sep 11 17:54:36.237051 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:54:36.237033    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-5xxdg], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-2mfqq" podUID="ece5ee46-fd0e-4be0-853e-4c40d9ea0f02"
Sep 11 17:54:36.263926 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.263891    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xxdg\" (UniqueName: \"kubernetes.io/projected/ece5ee46-fd0e-4be0-853e-4c40d9ea0f02-kube-api-access-5xxdg\") pod \"authorino-557f876f69-2mfqq\" (UID: \"ece5ee46-fd0e-4be0-853e-4c40d9ea0f02\") " pod="kuadrant-system/authorino-557f876f69-2mfqq"
Sep 11 17:54:36.302378 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.302330    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-d456r"]
Sep 11 17:54:36.305647 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.305630    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-d456r"
Sep 11 17:54:36.308712 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.308691    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 17:54:36.329925 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.329895    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-d456r"]
Sep 11 17:54:36.365369 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.365285    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/083bba1b-7b13-4677-9079-22821db5c838-tls-cert\") pod \"authorino-6d4dfc649f-d456r\" (UID: \"083bba1b-7b13-4677-9079-22821db5c838\") " pod="kuadrant-system/authorino-6d4dfc649f-d456r"
Sep 11 17:54:36.365369 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.365367    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsdbp\" (UniqueName: \"kubernetes.io/projected/083bba1b-7b13-4677-9079-22821db5c838-kube-api-access-tsdbp\") pod \"authorino-6d4dfc649f-d456r\" (UID: \"083bba1b-7b13-4677-9079-22821db5c838\") " pod="kuadrant-system/authorino-6d4dfc649f-d456r"
Sep 11 17:54:36.365604 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.365423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xxdg\" (UniqueName: \"kubernetes.io/projected/ece5ee46-fd0e-4be0-853e-4c40d9ea0f02-kube-api-access-5xxdg\") pod \"authorino-557f876f69-2mfqq\" (UID: \"ece5ee46-fd0e-4be0-853e-4c40d9ea0f02\") " pod="kuadrant-system/authorino-557f876f69-2mfqq"
Sep 11 17:54:36.378870 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.378782    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xxdg\" (UniqueName: \"kubernetes.io/projected/ece5ee46-fd0e-4be0-853e-4c40d9ea0f02-kube-api-access-5xxdg\") pod \"authorino-557f876f69-2mfqq\" (UID: \"ece5ee46-fd0e-4be0-853e-4c40d9ea0f02\") " pod="kuadrant-system/authorino-557f876f69-2mfqq"
Sep 11 17:54:36.413077 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.413046    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-d456r"]
Sep 11 17:54:36.413332 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:54:36.413315    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-tsdbp tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-d456r" podUID="083bba1b-7b13-4677-9079-22821db5c838"
Sep 11 17:54:36.451034 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.451000    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7d6bf4f45f-28jcd"]
Sep 11 17:54:36.454465 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.454447    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d6bf4f45f-28jcd"
Sep 11 17:54:36.465395 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.465370    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d6bf4f45f-28jcd"]
Sep 11 17:54:36.466272 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.466244    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/083bba1b-7b13-4677-9079-22821db5c838-tls-cert\") pod \"authorino-6d4dfc649f-d456r\" (UID: \"083bba1b-7b13-4677-9079-22821db5c838\") " pod="kuadrant-system/authorino-6d4dfc649f-d456r"
Sep 11 17:54:36.466400 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.466283    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsdbp\" (UniqueName: \"kubernetes.io/projected/083bba1b-7b13-4677-9079-22821db5c838-kube-api-access-tsdbp\") pod \"authorino-6d4dfc649f-d456r\" (UID: \"083bba1b-7b13-4677-9079-22821db5c838\") " pod="kuadrant-system/authorino-6d4dfc649f-d456r"
Sep 11 17:54:36.468678 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.468656    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/083bba1b-7b13-4677-9079-22821db5c838-tls-cert\") pod \"authorino-6d4dfc649f-d456r\" (UID: \"083bba1b-7b13-4677-9079-22821db5c838\") " pod="kuadrant-system/authorino-6d4dfc649f-d456r"
Sep 11 17:54:36.479591 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.479561    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsdbp\" (UniqueName: \"kubernetes.io/projected/083bba1b-7b13-4677-9079-22821db5c838-kube-api-access-tsdbp\") pod \"authorino-6d4dfc649f-d456r\" (UID: \"083bba1b-7b13-4677-9079-22821db5c838\") " pod="kuadrant-system/authorino-6d4dfc649f-d456r"
Sep 11 17:54:36.567674 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.567638    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c0de15cb-cdd0-4e2f-ad89-635d4046d9fe-tls-cert\") pod \"authorino-7d6bf4f45f-28jcd\" (UID: \"c0de15cb-cdd0-4e2f-ad89-635d4046d9fe\") " pod="kuadrant-system/authorino-7d6bf4f45f-28jcd"
Sep 11 17:54:36.567674 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.567674    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgg5w\" (UniqueName: \"kubernetes.io/projected/c0de15cb-cdd0-4e2f-ad89-635d4046d9fe-kube-api-access-tgg5w\") pod \"authorino-7d6bf4f45f-28jcd\" (UID: \"c0de15cb-cdd0-4e2f-ad89-635d4046d9fe\") " pod="kuadrant-system/authorino-7d6bf4f45f-28jcd"
Sep 11 17:54:36.669093 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.669064    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c0de15cb-cdd0-4e2f-ad89-635d4046d9fe-tls-cert\") pod \"authorino-7d6bf4f45f-28jcd\" (UID: \"c0de15cb-cdd0-4e2f-ad89-635d4046d9fe\") " pod="kuadrant-system/authorino-7d6bf4f45f-28jcd"
Sep 11 17:54:36.669246 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.669100    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgg5w\" (UniqueName: \"kubernetes.io/projected/c0de15cb-cdd0-4e2f-ad89-635d4046d9fe-kube-api-access-tgg5w\") pod \"authorino-7d6bf4f45f-28jcd\" (UID: \"c0de15cb-cdd0-4e2f-ad89-635d4046d9fe\") " pod="kuadrant-system/authorino-7d6bf4f45f-28jcd"
Sep 11 17:54:36.671510 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.671484    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c0de15cb-cdd0-4e2f-ad89-635d4046d9fe-tls-cert\") pod \"authorino-7d6bf4f45f-28jcd\" (UID: \"c0de15cb-cdd0-4e2f-ad89-635d4046d9fe\") " pod="kuadrant-system/authorino-7d6bf4f45f-28jcd"
Sep 11 17:54:36.681433 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.681407    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgg5w\" (UniqueName: \"kubernetes.io/projected/c0de15cb-cdd0-4e2f-ad89-635d4046d9fe-kube-api-access-tgg5w\") pod \"authorino-7d6bf4f45f-28jcd\" (UID: \"c0de15cb-cdd0-4e2f-ad89-635d4046d9fe\") " pod="kuadrant-system/authorino-7d6bf4f45f-28jcd"
Sep 11 17:54:36.764497 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.764463    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d6bf4f45f-28jcd"
Sep 11 17:54:36.793019 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.792992    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-d456r"
Sep 11 17:54:36.793019 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.793006    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-2mfqq"
Sep 11 17:54:36.800525 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.800502    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-d456r"
Sep 11 17:54:36.803522 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.803504    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-2mfqq"
Sep 11 17:54:36.872517 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.872490    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/083bba1b-7b13-4677-9079-22821db5c838-tls-cert\") pod \"083bba1b-7b13-4677-9079-22821db5c838\" (UID: \"083bba1b-7b13-4677-9079-22821db5c838\") "
Sep 11 17:54:36.872517 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.872520    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsdbp\" (UniqueName: \"kubernetes.io/projected/083bba1b-7b13-4677-9079-22821db5c838-kube-api-access-tsdbp\") pod \"083bba1b-7b13-4677-9079-22821db5c838\" (UID: \"083bba1b-7b13-4677-9079-22821db5c838\") "
Sep 11 17:54:36.872766 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.872581    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xxdg\" (UniqueName: \"kubernetes.io/projected/ece5ee46-fd0e-4be0-853e-4c40d9ea0f02-kube-api-access-5xxdg\") pod \"ece5ee46-fd0e-4be0-853e-4c40d9ea0f02\" (UID: \"ece5ee46-fd0e-4be0-853e-4c40d9ea0f02\") "
Sep 11 17:54:36.874776 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.874749    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/083bba1b-7b13-4677-9079-22821db5c838-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "083bba1b-7b13-4677-9079-22821db5c838" (UID: "083bba1b-7b13-4677-9079-22821db5c838"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:54:36.874776 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.874748    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083bba1b-7b13-4677-9079-22821db5c838-kube-api-access-tsdbp" (OuterVolumeSpecName: "kube-api-access-tsdbp") pod "083bba1b-7b13-4677-9079-22821db5c838" (UID: "083bba1b-7b13-4677-9079-22821db5c838"). InnerVolumeSpecName "kube-api-access-tsdbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:54:36.874931 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.874785    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece5ee46-fd0e-4be0-853e-4c40d9ea0f02-kube-api-access-5xxdg" (OuterVolumeSpecName: "kube-api-access-5xxdg") pod "ece5ee46-fd0e-4be0-853e-4c40d9ea0f02" (UID: "ece5ee46-fd0e-4be0-853e-4c40d9ea0f02"). InnerVolumeSpecName "kube-api-access-5xxdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:54:36.973718 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.973642    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xxdg\" (UniqueName: \"kubernetes.io/projected/ece5ee46-fd0e-4be0-853e-4c40d9ea0f02-kube-api-access-5xxdg\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:54:36.973718 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.973672    2559 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/083bba1b-7b13-4677-9079-22821db5c838-tls-cert\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:54:36.973718 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:36.973682    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsdbp\" (UniqueName: \"kubernetes.io/projected/083bba1b-7b13-4677-9079-22821db5c838-kube-api-access-tsdbp\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:54:37.044447 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:37.044422    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d6bf4f45f-28jcd"]
Sep 11 17:54:37.046368 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:54:37.046325    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0de15cb_cdd0_4e2f_ad89_635d4046d9fe.slice/crio-ba3ea0e01938f8f4211526d6fc419832ca9225a0f599c5701c150201f3d2aee3 WatchSource:0}: Error finding container ba3ea0e01938f8f4211526d6fc419832ca9225a0f599c5701c150201f3d2aee3: Status 404 returned error can't find the container with id ba3ea0e01938f8f4211526d6fc419832ca9225a0f599c5701c150201f3d2aee3
Sep 11 17:54:37.800271 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:37.800242    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d6bf4f45f-28jcd" event={"ID":"c0de15cb-cdd0-4e2f-ad89-635d4046d9fe","Type":"ContainerStarted","Data":"8d19ff6adb405cf634993520ef5dbc17a88b3db40bf4dc93750a84f6aa9ca20b"}
Sep 11 17:54:37.800730 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:37.800285    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d6bf4f45f-28jcd" event={"ID":"c0de15cb-cdd0-4e2f-ad89-635d4046d9fe","Type":"ContainerStarted","Data":"ba3ea0e01938f8f4211526d6fc419832ca9225a0f599c5701c150201f3d2aee3"}
Sep 11 17:54:37.800730 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:37.800345    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-2mfqq"
Sep 11 17:54:37.800730 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:37.800381    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-d456r"
Sep 11 17:54:37.831506 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:37.831439    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7d6bf4f45f-28jcd" podStartSLOduration=1.158242295 podStartE2EDuration="1.831422083s" totalImagesPullingTime="673.179788ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:54:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:37.827972074 +0000 UTC m=+583.273072648" watchObservedRunningTime="2026-09-11 17:54:37.831422083 +0000 UTC m=+583.276522655"
Sep 11 17:54:37.868780 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:37.868752    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-2mfqq"]
Sep 11 17:54:37.878997 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:37.878967    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-2mfqq"]
Sep 11 17:54:37.896212 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:37.896177    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-7k4b4"]
Sep 11 17:54:37.896397 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:37.896376    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-7k4b4" podUID="89668824-6889-4b63-9cd2-ea109021d98c" containerName="authorino" containerID="cri-o://bb07500fd00977d203503c03b804d3163382762070b3a1280b7b69d175389f2d" gracePeriod=30
Sep 11 17:54:37.949503 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:37.949468    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-d456r"]
Sep 11 17:54:37.955927 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:37.955899    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-d456r"]
Sep 11 17:54:38.138598 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:38.138572    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-7k4b4"
Sep 11 17:54:38.285722 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:38.285640    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz4ml\" (UniqueName: \"kubernetes.io/projected/89668824-6889-4b63-9cd2-ea109021d98c-kube-api-access-dz4ml\") pod \"89668824-6889-4b63-9cd2-ea109021d98c\" (UID: \"89668824-6889-4b63-9cd2-ea109021d98c\") "
Sep 11 17:54:38.287736 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:38.287715    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89668824-6889-4b63-9cd2-ea109021d98c-kube-api-access-dz4ml" (OuterVolumeSpecName: "kube-api-access-dz4ml") pod "89668824-6889-4b63-9cd2-ea109021d98c" (UID: "89668824-6889-4b63-9cd2-ea109021d98c"). InnerVolumeSpecName "kube-api-access-dz4ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:54:38.386813 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:38.386787    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz4ml\" (UniqueName: \"kubernetes.io/projected/89668824-6889-4b63-9cd2-ea109021d98c-kube-api-access-dz4ml\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:54:38.807035 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:38.807004    2559 generic.go:356] "Generic (PLEG): container finished" podID="89668824-6889-4b63-9cd2-ea109021d98c" containerID="bb07500fd00977d203503c03b804d3163382762070b3a1280b7b69d175389f2d" exitCode=0
Sep 11 17:54:38.807508 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:38.807048    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-7k4b4"
Sep 11 17:54:38.807508 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:38.807101    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-7k4b4" event={"ID":"89668824-6889-4b63-9cd2-ea109021d98c","Type":"ContainerDied","Data":"bb07500fd00977d203503c03b804d3163382762070b3a1280b7b69d175389f2d"}
Sep 11 17:54:38.807508 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:38.807144    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-7k4b4" event={"ID":"89668824-6889-4b63-9cd2-ea109021d98c","Type":"ContainerDied","Data":"7e6da6c839c32c613f4f2f3e1b2303383fe5344c8b0ce1e666afd9cd71a185c6"}
Sep 11 17:54:38.807508 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:38.807166    2559 scope.go:117] "RemoveContainer" containerID="bb07500fd00977d203503c03b804d3163382762070b3a1280b7b69d175389f2d"
Sep 11 17:54:38.816301 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:38.816274    2559 scope.go:117] "RemoveContainer" containerID="bb07500fd00977d203503c03b804d3163382762070b3a1280b7b69d175389f2d"
Sep 11 17:54:38.816614 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:54:38.816596    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb07500fd00977d203503c03b804d3163382762070b3a1280b7b69d175389f2d\": container with ID starting with bb07500fd00977d203503c03b804d3163382762070b3a1280b7b69d175389f2d not found: ID does not exist" containerID="bb07500fd00977d203503c03b804d3163382762070b3a1280b7b69d175389f2d"
Sep 11 17:54:38.816662 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:38.816623    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb07500fd00977d203503c03b804d3163382762070b3a1280b7b69d175389f2d"} err="failed to get container status \"bb07500fd00977d203503c03b804d3163382762070b3a1280b7b69d175389f2d\": rpc error: code = NotFound desc = could not find container \"bb07500fd00977d203503c03b804d3163382762070b3a1280b7b69d175389f2d\": container with ID starting with bb07500fd00977d203503c03b804d3163382762070b3a1280b7b69d175389f2d not found: ID does not exist"
Sep 11 17:54:38.831453 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:38.831426    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-7k4b4"]
Sep 11 17:54:38.834464 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:38.834440    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-7k4b4"]
Sep 11 17:54:39.056377 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:39.054189    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="083bba1b-7b13-4677-9079-22821db5c838" path="/var/lib/kubelet/pods/083bba1b-7b13-4677-9079-22821db5c838/volumes"
Sep 11 17:54:39.056377 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:39.054614    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89668824-6889-4b63-9cd2-ea109021d98c" path="/var/lib/kubelet/pods/89668824-6889-4b63-9cd2-ea109021d98c/volumes"
Sep 11 17:54:39.056377 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:54:39.055072    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ece5ee46-fd0e-4be0-853e-4c40d9ea0f02" path="/var/lib/kubelet/pods/ece5ee46-fd0e-4be0-853e-4c40d9ea0f02/volumes"
Sep 11 17:55:22.542045 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:55:22.541962    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 17:55:42.770562 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:55:42.770516    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 17:55:52.845723 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:55:52.845683    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 17:55:58.940050 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:55:58.940011    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 17:56:08.643791 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:08.643750    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 17:56:17.885838 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:17.885804    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"]
Sep 11 17:56:17.886333 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:17.886309    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89668824-6889-4b63-9cd2-ea109021d98c" containerName="authorino"
Sep 11 17:56:17.886453 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:17.886337    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="89668824-6889-4b63-9cd2-ea109021d98c" containerName="authorino"
Sep 11 17:56:17.886509 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:17.886482    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="89668824-6889-4b63-9cd2-ea109021d98c" containerName="authorino"
Sep 11 17:56:17.889668 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:17.889647    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"
Sep 11 17:56:17.893178 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:17.893157    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 11 17:56:17.893178 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:17.893168    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jwbsk\""
Sep 11 17:56:17.893575 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:17.893549    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 17:56:17.893675 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:17.893663    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 17:56:17.901472 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:17.901446    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8282f8da-b17f-4872-b70e-391487cbefee-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x\" (UID: \"8282f8da-b17f-4872-b70e-391487cbefee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"
Sep 11 17:56:17.901579 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:17.901508    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8282f8da-b17f-4872-b70e-391487cbefee-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x\" (UID: \"8282f8da-b17f-4872-b70e-391487cbefee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"
Sep 11 17:56:17.901579 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:17.901557    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8282f8da-b17f-4872-b70e-391487cbefee-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x\" (UID: \"8282f8da-b17f-4872-b70e-391487cbefee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"
Sep 11 17:56:17.901651 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:17.901608    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8282f8da-b17f-4872-b70e-391487cbefee-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x\" (UID: \"8282f8da-b17f-4872-b70e-391487cbefee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"
Sep 11 17:56:17.901651 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:17.901624    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6dmk\" (UniqueName: \"kubernetes.io/projected/8282f8da-b17f-4872-b70e-391487cbefee-kube-api-access-t6dmk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x\" (UID: \"8282f8da-b17f-4872-b70e-391487cbefee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"
Sep 11 17:56:17.901723 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:17.901654    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8282f8da-b17f-4872-b70e-391487cbefee-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x\" (UID: \"8282f8da-b17f-4872-b70e-391487cbefee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"
Sep 11 17:56:17.976543 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:17.976510    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"]
Sep 11 17:56:18.002205 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:18.002171    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8282f8da-b17f-4872-b70e-391487cbefee-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x\" (UID: \"8282f8da-b17f-4872-b70e-391487cbefee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"
Sep 11 17:56:18.002397 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:18.002222    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8282f8da-b17f-4872-b70e-391487cbefee-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x\" (UID: \"8282f8da-b17f-4872-b70e-391487cbefee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"
Sep 11 17:56:18.002397 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:18.002261    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8282f8da-b17f-4872-b70e-391487cbefee-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x\" (UID: \"8282f8da-b17f-4872-b70e-391487cbefee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"
Sep 11 17:56:18.002397 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:18.002281    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8282f8da-b17f-4872-b70e-391487cbefee-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x\" (UID: \"8282f8da-b17f-4872-b70e-391487cbefee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"
Sep 11 17:56:18.002397 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:18.002298    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6dmk\" (UniqueName: \"kubernetes.io/projected/8282f8da-b17f-4872-b70e-391487cbefee-kube-api-access-t6dmk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x\" (UID: \"8282f8da-b17f-4872-b70e-391487cbefee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"
Sep 11 17:56:18.002397 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:18.002326    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8282f8da-b17f-4872-b70e-391487cbefee-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x\" (UID: \"8282f8da-b17f-4872-b70e-391487cbefee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"
Sep 11 17:56:18.002707 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:18.002683    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8282f8da-b17f-4872-b70e-391487cbefee-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x\" (UID: \"8282f8da-b17f-4872-b70e-391487cbefee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"
Sep 11 17:56:18.002753 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:18.002703    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8282f8da-b17f-4872-b70e-391487cbefee-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x\" (UID: \"8282f8da-b17f-4872-b70e-391487cbefee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"
Sep 11 17:56:18.002805 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:18.002791    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8282f8da-b17f-4872-b70e-391487cbefee-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x\" (UID: \"8282f8da-b17f-4872-b70e-391487cbefee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"
Sep 11 17:56:18.004624 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:18.004599    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8282f8da-b17f-4872-b70e-391487cbefee-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x\" (UID: \"8282f8da-b17f-4872-b70e-391487cbefee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"
Sep 11 17:56:18.004866 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:18.004852    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8282f8da-b17f-4872-b70e-391487cbefee-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x\" (UID: \"8282f8da-b17f-4872-b70e-391487cbefee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"
Sep 11 17:56:18.016287 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:18.016262    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6dmk\" (UniqueName: \"kubernetes.io/projected/8282f8da-b17f-4872-b70e-391487cbefee-kube-api-access-t6dmk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x\" (UID: \"8282f8da-b17f-4872-b70e-391487cbefee\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"
Sep 11 17:56:18.200012 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:18.199921    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"
Sep 11 17:56:18.484844 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:18.484809    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"]
Sep 11 17:56:18.485993 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:56:18.485964    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8282f8da_b17f_4872_b70e_391487cbefee.slice/crio-89cf2c4c06c4e55125f1ec7ebdad9c900bbd658a01afb0397f7c47c4a5bb75da WatchSource:0}: Error finding container 89cf2c4c06c4e55125f1ec7ebdad9c900bbd658a01afb0397f7c47c4a5bb75da: Status 404 returned error can't find the container with id 89cf2c4c06c4e55125f1ec7ebdad9c900bbd658a01afb0397f7c47c4a5bb75da
Sep 11 17:56:18.487372 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:18.487339    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:56:19.315308 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:19.315265    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x" event={"ID":"8282f8da-b17f-4872-b70e-391487cbefee","Type":"ContainerStarted","Data":"89cf2c4c06c4e55125f1ec7ebdad9c900bbd658a01afb0397f7c47c4a5bb75da"}
Sep 11 17:56:19.441607 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:19.441563    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 17:56:20.321532 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:20.321486    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x" event={"ID":"8282f8da-b17f-4872-b70e-391487cbefee","Type":"ContainerStarted","Data":"5d2aafb6dc46bffe578d2e3e13282e107ed06336fd057b82271b2a34af037f25"}
Sep 11 17:56:20.358220 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:20.358164    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x" podStartSLOduration=2.348627819 podStartE2EDuration="3.358146277s" totalImagesPullingTime="1.009518458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:56:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:56:20.355050257 +0000 UTC m=+685.800150828" watchObservedRunningTime="2026-09-11 17:56:20.358146277 +0000 UTC m=+685.803246847"
Sep 11 17:56:25.343627 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:25.343592    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 17:56:28.200597 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:28.200550    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"
Sep 11 17:56:28.201056 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:28.200635    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"
Sep 11 17:56:28.213165 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:28.213137    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"
Sep 11 17:56:28.376118 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:28.376088    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x"
Sep 11 17:56:32.957403 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:32.957345    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"]
Sep 11 17:56:32.960971 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:32.960949    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"
Sep 11 17:56:32.963764 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:32.963741    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 17:56:32.984136 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:32.984103    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"]
Sep 11 17:56:33.031035 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:33.030990    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f581ce8c-7f44-478d-8936-37010f86a176-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-csbfp\" (UID: \"f581ce8c-7f44-478d-8936-37010f86a176\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"
Sep 11 17:56:33.031212 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:33.031052    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f581ce8c-7f44-478d-8936-37010f86a176-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-csbfp\" (UID: \"f581ce8c-7f44-478d-8936-37010f86a176\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"
Sep 11 17:56:33.031212 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:33.031125    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f581ce8c-7f44-478d-8936-37010f86a176-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-csbfp\" (UID: \"f581ce8c-7f44-478d-8936-37010f86a176\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"
Sep 11 17:56:33.031212 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:33.031155    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnf42\" (UniqueName: \"kubernetes.io/projected/f581ce8c-7f44-478d-8936-37010f86a176-kube-api-access-qnf42\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-csbfp\" (UID: \"f581ce8c-7f44-478d-8936-37010f86a176\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"
Sep 11 17:56:33.031212 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:33.031183    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f581ce8c-7f44-478d-8936-37010f86a176-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-csbfp\" (UID: \"f581ce8c-7f44-478d-8936-37010f86a176\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"
Sep 11 17:56:33.031212 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:33.031201    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f581ce8c-7f44-478d-8936-37010f86a176-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-csbfp\" (UID: \"f581ce8c-7f44-478d-8936-37010f86a176\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"
Sep 11 17:56:33.132108 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:33.132061    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnf42\" (UniqueName: \"kubernetes.io/projected/f581ce8c-7f44-478d-8936-37010f86a176-kube-api-access-qnf42\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-csbfp\" (UID: \"f581ce8c-7f44-478d-8936-37010f86a176\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"
Sep 11 17:56:33.132303 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:33.132124    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f581ce8c-7f44-478d-8936-37010f86a176-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-csbfp\" (UID: \"f581ce8c-7f44-478d-8936-37010f86a176\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"
Sep 11 17:56:33.132303 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:33.132150    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f581ce8c-7f44-478d-8936-37010f86a176-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-csbfp\" (UID: \"f581ce8c-7f44-478d-8936-37010f86a176\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"
Sep 11 17:56:33.132303 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:33.132187    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f581ce8c-7f44-478d-8936-37010f86a176-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-csbfp\" (UID: \"f581ce8c-7f44-478d-8936-37010f86a176\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"
Sep 11 17:56:33.132303 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:33.132250    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f581ce8c-7f44-478d-8936-37010f86a176-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-csbfp\" (UID: \"f581ce8c-7f44-478d-8936-37010f86a176\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"
Sep 11 17:56:33.132303 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:33.132303    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f581ce8c-7f44-478d-8936-37010f86a176-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-csbfp\" (UID: \"f581ce8c-7f44-478d-8936-37010f86a176\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"
Sep 11 17:56:33.132623 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:33.132600    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f581ce8c-7f44-478d-8936-37010f86a176-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-csbfp\" (UID: \"f581ce8c-7f44-478d-8936-37010f86a176\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"
Sep 11 17:56:33.132751 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:33.132730    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f581ce8c-7f44-478d-8936-37010f86a176-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-csbfp\" (UID: \"f581ce8c-7f44-478d-8936-37010f86a176\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"
Sep 11 17:56:33.132796 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:33.132762    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f581ce8c-7f44-478d-8936-37010f86a176-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-csbfp\" (UID: \"f581ce8c-7f44-478d-8936-37010f86a176\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"
Sep 11 17:56:33.134605 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:33.134573    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f581ce8c-7f44-478d-8936-37010f86a176-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-csbfp\" (UID: \"f581ce8c-7f44-478d-8936-37010f86a176\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"
Sep 11 17:56:33.134830 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:33.134810    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f581ce8c-7f44-478d-8936-37010f86a176-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-csbfp\" (UID: \"f581ce8c-7f44-478d-8936-37010f86a176\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"
Sep 11 17:56:33.144776 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:33.144755    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnf42\" (UniqueName: \"kubernetes.io/projected/f581ce8c-7f44-478d-8936-37010f86a176-kube-api-access-qnf42\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-csbfp\" (UID: \"f581ce8c-7f44-478d-8936-37010f86a176\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"
Sep 11 17:56:33.273741 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:33.273643    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"
Sep 11 17:56:33.552731 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:33.552699    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"]
Sep 11 17:56:33.554174 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:56:33.554145    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf581ce8c_7f44_478d_8936_37010f86a176.slice/crio-5cebb6ac167a8dbe0870ff50bfd451b36b6ef385bc94cae053f90e7141a28caa WatchSource:0}: Error finding container 5cebb6ac167a8dbe0870ff50bfd451b36b6ef385bc94cae053f90e7141a28caa: Status 404 returned error can't find the container with id 5cebb6ac167a8dbe0870ff50bfd451b36b6ef385bc94cae053f90e7141a28caa
Sep 11 17:56:34.397271 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:34.397232    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp" event={"ID":"f581ce8c-7f44-478d-8936-37010f86a176","Type":"ContainerStarted","Data":"78e2fd08e891238dfc469b5b1fdb5f5c2b438769c6e181cf5012ae826a9d3bb7"}
Sep 11 17:56:34.397271 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:34.397277    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp" event={"ID":"f581ce8c-7f44-478d-8936-37010f86a176","Type":"ContainerStarted","Data":"5cebb6ac167a8dbe0870ff50bfd451b36b6ef385bc94cae053f90e7141a28caa"}
Sep 11 17:56:35.256085 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:35.256027    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp" podStartSLOduration=3.082102872 podStartE2EDuration="3.256007632s" totalImagesPullingTime="173.90476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:56:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:56:34.430933598 +0000 UTC m=+699.876034169" watchObservedRunningTime="2026-09-11 17:56:35.256007632 +0000 UTC m=+700.701108205"
Sep 11 17:56:35.257754 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:35.257733    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 17:56:43.274301 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:43.274267    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"
Sep 11 17:56:43.274892 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:43.274322    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"
Sep 11 17:56:43.287065 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:43.287033    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"
Sep 11 17:56:43.453929 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:56:43.453897    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-csbfp"
Sep 11 17:57:38.672320 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:38.672275    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d6dff65c8-qq67f"]
Sep 11 17:57:38.675689 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:38.675668    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d6dff65c8-qq67f"
Sep 11 17:57:38.688269 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:38.688243    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d6dff65c8-qq67f"]
Sep 11 17:57:38.840300 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:38.840251    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/088960ae-3c43-479f-b684-0087be132b13-tls-cert\") pod \"authorino-5d6dff65c8-qq67f\" (UID: \"088960ae-3c43-479f-b684-0087be132b13\") " pod="kuadrant-system/authorino-5d6dff65c8-qq67f"
Sep 11 17:57:38.840533 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:38.840442    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7kgb\" (UniqueName: \"kubernetes.io/projected/088960ae-3c43-479f-b684-0087be132b13-kube-api-access-f7kgb\") pod \"authorino-5d6dff65c8-qq67f\" (UID: \"088960ae-3c43-479f-b684-0087be132b13\") " pod="kuadrant-system/authorino-5d6dff65c8-qq67f"
Sep 11 17:57:38.941770 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:38.941668    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/088960ae-3c43-479f-b684-0087be132b13-tls-cert\") pod \"authorino-5d6dff65c8-qq67f\" (UID: \"088960ae-3c43-479f-b684-0087be132b13\") " pod="kuadrant-system/authorino-5d6dff65c8-qq67f"
Sep 11 17:57:38.941955 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:38.941791    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7kgb\" (UniqueName: \"kubernetes.io/projected/088960ae-3c43-479f-b684-0087be132b13-kube-api-access-f7kgb\") pod \"authorino-5d6dff65c8-qq67f\" (UID: \"088960ae-3c43-479f-b684-0087be132b13\") " pod="kuadrant-system/authorino-5d6dff65c8-qq67f"
Sep 11 17:57:38.944085 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:38.944055    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/088960ae-3c43-479f-b684-0087be132b13-tls-cert\") pod \"authorino-5d6dff65c8-qq67f\" (UID: \"088960ae-3c43-479f-b684-0087be132b13\") " pod="kuadrant-system/authorino-5d6dff65c8-qq67f"
Sep 11 17:57:38.950862 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:38.950837    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7kgb\" (UniqueName: \"kubernetes.io/projected/088960ae-3c43-479f-b684-0087be132b13-kube-api-access-f7kgb\") pod \"authorino-5d6dff65c8-qq67f\" (UID: \"088960ae-3c43-479f-b684-0087be132b13\") " pod="kuadrant-system/authorino-5d6dff65c8-qq67f"
Sep 11 17:57:38.984615 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:38.984577    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d6dff65c8-qq67f"
Sep 11 17:57:39.260817 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:39.260784    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d6dff65c8-qq67f"]
Sep 11 17:57:39.262665 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:57:39.262638    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod088960ae_3c43_479f_b684_0087be132b13.slice/crio-9553764b0466998c963468652c0809e92e0a3f5ff7d22eb9b99136a8a4e6163b WatchSource:0}: Error finding container 9553764b0466998c963468652c0809e92e0a3f5ff7d22eb9b99136a8a4e6163b: Status 404 returned error can't find the container with id 9553764b0466998c963468652c0809e92e0a3f5ff7d22eb9b99136a8a4e6163b
Sep 11 17:57:39.743198 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:39.743164    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d6dff65c8-qq67f" event={"ID":"088960ae-3c43-479f-b684-0087be132b13","Type":"ContainerStarted","Data":"9553764b0466998c963468652c0809e92e0a3f5ff7d22eb9b99136a8a4e6163b"}
Sep 11 17:57:40.752229 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:40.752185    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d6dff65c8-qq67f" event={"ID":"088960ae-3c43-479f-b684-0087be132b13","Type":"ContainerStarted","Data":"963e6d9fa1fb33ac29c172f17de912a2262297f415d14841276c943de5138d45"}
Sep 11 17:57:40.811366 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:40.811293    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d6dff65c8-qq67f" podStartSLOduration=2.065965272 podStartE2EDuration="2.811273912s" totalImagesPullingTime="745.30864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:57:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:57:40.792098266 +0000 UTC m=+766.237198837" watchObservedRunningTime="2026-09-11 17:57:40.811273912 +0000 UTC m=+766.256374482"
Sep 11 17:57:40.861921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:40.861887    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d6bf4f45f-28jcd"]
Sep 11 17:57:40.862082 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:40.862058    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7d6bf4f45f-28jcd" podUID="c0de15cb-cdd0-4e2f-ad89-635d4046d9fe" containerName="authorino" containerID="cri-o://8d19ff6adb405cf634993520ef5dbc17a88b3db40bf4dc93750a84f6aa9ca20b" gracePeriod=30
Sep 11 17:57:41.114189 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:41.114149    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d6bf4f45f-28jcd"
Sep 11 17:57:41.264980 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:41.264937    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgg5w\" (UniqueName: \"kubernetes.io/projected/c0de15cb-cdd0-4e2f-ad89-635d4046d9fe-kube-api-access-tgg5w\") pod \"c0de15cb-cdd0-4e2f-ad89-635d4046d9fe\" (UID: \"c0de15cb-cdd0-4e2f-ad89-635d4046d9fe\") "
Sep 11 17:57:41.265170 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:41.265016    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c0de15cb-cdd0-4e2f-ad89-635d4046d9fe-tls-cert\") pod \"c0de15cb-cdd0-4e2f-ad89-635d4046d9fe\" (UID: \"c0de15cb-cdd0-4e2f-ad89-635d4046d9fe\") "
Sep 11 17:57:41.267099 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:41.267064    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0de15cb-cdd0-4e2f-ad89-635d4046d9fe-kube-api-access-tgg5w" (OuterVolumeSpecName: "kube-api-access-tgg5w") pod "c0de15cb-cdd0-4e2f-ad89-635d4046d9fe" (UID: "c0de15cb-cdd0-4e2f-ad89-635d4046d9fe"). InnerVolumeSpecName "kube-api-access-tgg5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:57:41.276132 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:41.276067    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0de15cb-cdd0-4e2f-ad89-635d4046d9fe-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "c0de15cb-cdd0-4e2f-ad89-635d4046d9fe" (UID: "c0de15cb-cdd0-4e2f-ad89-635d4046d9fe"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:57:41.366732 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:41.366684    2559 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c0de15cb-cdd0-4e2f-ad89-635d4046d9fe-tls-cert\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:57:41.366732 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:41.366732    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgg5w\" (UniqueName: \"kubernetes.io/projected/c0de15cb-cdd0-4e2f-ad89-635d4046d9fe-kube-api-access-tgg5w\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:57:41.759574 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:41.759536    2559 generic.go:356] "Generic (PLEG): container finished" podID="c0de15cb-cdd0-4e2f-ad89-635d4046d9fe" containerID="8d19ff6adb405cf634993520ef5dbc17a88b3db40bf4dc93750a84f6aa9ca20b" exitCode=0
Sep 11 17:57:41.760060 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:41.759579    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d6bf4f45f-28jcd"
Sep 11 17:57:41.760060 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:41.759624    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d6bf4f45f-28jcd" event={"ID":"c0de15cb-cdd0-4e2f-ad89-635d4046d9fe","Type":"ContainerDied","Data":"8d19ff6adb405cf634993520ef5dbc17a88b3db40bf4dc93750a84f6aa9ca20b"}
Sep 11 17:57:41.760060 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:41.759674    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d6bf4f45f-28jcd" event={"ID":"c0de15cb-cdd0-4e2f-ad89-635d4046d9fe","Type":"ContainerDied","Data":"ba3ea0e01938f8f4211526d6fc419832ca9225a0f599c5701c150201f3d2aee3"}
Sep 11 17:57:41.760060 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:41.759703    2559 scope.go:117] "RemoveContainer" containerID="8d19ff6adb405cf634993520ef5dbc17a88b3db40bf4dc93750a84f6aa9ca20b"
Sep 11 17:57:41.769116 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:41.769092    2559 scope.go:117] "RemoveContainer" containerID="8d19ff6adb405cf634993520ef5dbc17a88b3db40bf4dc93750a84f6aa9ca20b"
Sep 11 17:57:41.769487 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:57:41.769465    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d19ff6adb405cf634993520ef5dbc17a88b3db40bf4dc93750a84f6aa9ca20b\": container with ID starting with 8d19ff6adb405cf634993520ef5dbc17a88b3db40bf4dc93750a84f6aa9ca20b not found: ID does not exist" containerID="8d19ff6adb405cf634993520ef5dbc17a88b3db40bf4dc93750a84f6aa9ca20b"
Sep 11 17:57:41.769568 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:41.769497    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d19ff6adb405cf634993520ef5dbc17a88b3db40bf4dc93750a84f6aa9ca20b"} err="failed to get container status \"8d19ff6adb405cf634993520ef5dbc17a88b3db40bf4dc93750a84f6aa9ca20b\": rpc error: code = NotFound desc = could not find container \"8d19ff6adb405cf634993520ef5dbc17a88b3db40bf4dc93750a84f6aa9ca20b\": container with ID starting with 8d19ff6adb405cf634993520ef5dbc17a88b3db40bf4dc93750a84f6aa9ca20b not found: ID does not exist"
Sep 11 17:57:41.793729 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:41.793691    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d6bf4f45f-28jcd"]
Sep 11 17:57:41.796941 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:41.796911    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7d6bf4f45f-28jcd"]
Sep 11 17:57:43.053544 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:57:43.053503    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0de15cb-cdd0-4e2f-ad89-635d4046d9fe" path="/var/lib/kubelet/pods/c0de15cb-cdd0-4e2f-ad89-635d4046d9fe/volumes"
Sep 11 17:58:01.229052 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:01.229016    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7874-789f10"]
Sep 11 17:58:01.229466 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:01.229391    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0de15cb-cdd0-4e2f-ad89-635d4046d9fe" containerName="authorino"
Sep 11 17:58:01.229466 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:01.229404    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0de15cb-cdd0-4e2f-ad89-635d4046d9fe" containerName="authorino"
Sep 11 17:58:01.229541 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:01.229469    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0de15cb-cdd0-4e2f-ad89-635d4046d9fe" containerName="authorino"
Sep 11 17:58:01.233736 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:01.233718    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7874-789f10"
Sep 11 17:58:01.237520 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:01.237500    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nbc6p\""
Sep 11 17:58:01.253588 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:01.253556    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7874-789f10"]
Sep 11 17:58:01.345723 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:01.345687    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f995\" (UniqueName: \"kubernetes.io/projected/667ea483-c5e1-4c1c-943d-e1325c0d510b-kube-api-access-9f995\") pod \"test-curl-7874-789f10\" (UID: \"667ea483-c5e1-4c1c-943d-e1325c0d510b\") " pod="opendatahub/test-curl-7874-789f10"
Sep 11 17:58:01.446858 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:01.446811    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f995\" (UniqueName: \"kubernetes.io/projected/667ea483-c5e1-4c1c-943d-e1325c0d510b-kube-api-access-9f995\") pod \"test-curl-7874-789f10\" (UID: \"667ea483-c5e1-4c1c-943d-e1325c0d510b\") " pod="opendatahub/test-curl-7874-789f10"
Sep 11 17:58:01.455908 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:01.455876    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f995\" (UniqueName: \"kubernetes.io/projected/667ea483-c5e1-4c1c-943d-e1325c0d510b-kube-api-access-9f995\") pod \"test-curl-7874-789f10\" (UID: \"667ea483-c5e1-4c1c-943d-e1325c0d510b\") " pod="opendatahub/test-curl-7874-789f10"
Sep 11 17:58:01.543033 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:01.542950    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7874-789f10"
Sep 11 17:58:01.818775 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:01.818753    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7874-789f10"]
Sep 11 17:58:01.820864 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:58:01.820831    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod667ea483_c5e1_4c1c_943d_e1325c0d510b.slice/crio-274e3950d73b5323f3307ee1ebcd2b6e22c6450923b1230d5bd72d08dc6418b5 WatchSource:0}: Error finding container 274e3950d73b5323f3307ee1ebcd2b6e22c6450923b1230d5bd72d08dc6418b5: Status 404 returned error can't find the container with id 274e3950d73b5323f3307ee1ebcd2b6e22c6450923b1230d5bd72d08dc6418b5
Sep 11 17:58:01.865573 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:01.865536    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7874-789f10" event={"ID":"667ea483-c5e1-4c1c-943d-e1325c0d510b","Type":"ContainerStarted","Data":"274e3950d73b5323f3307ee1ebcd2b6e22c6450923b1230d5bd72d08dc6418b5"}
Sep 11 17:58:02.712705 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:02.712664    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 17:58:04.898016 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:04.897976    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7874-789f10" event={"ID":"667ea483-c5e1-4c1c-943d-e1325c0d510b","Type":"ContainerStarted","Data":"a34bfe6a8fafe41bcad09f00859f7a45aca1bbe6b25c1c16e98a783fe9007715"}
Sep 11 17:58:04.922260 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:04.922210    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7874-789f10" podStartSLOduration=1.60839754 podStartE2EDuration="3.922194795s" totalImagesPullingTime="2.313797255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:58:04.921446292 +0000 UTC m=+790.366546862" watchObservedRunningTime="2026-09-11 17:58:04.922194795 +0000 UTC m=+790.367295365"
Sep 11 17:58:05.798015 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:05.797917    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7874-789f10"]
Sep 11 17:58:05.804733 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:05.804705    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7874-789f10"]
Sep 11 17:58:05.904195 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:58:05.904163    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7874-789f10" container="test-curl-7874-789f10"
Sep 11 17:58:05.941919 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:05.941885    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7874-bae2f4"]
Sep 11 17:58:05.942243 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:05.942229    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="667ea483-c5e1-4c1c-943d-e1325c0d510b" containerName="test-curl-7874-789f10"
Sep 11 17:58:05.942292 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:05.942244    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="667ea483-c5e1-4c1c-943d-e1325c0d510b" containerName="test-curl-7874-789f10"
Sep 11 17:58:05.942330 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:05.942318    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="667ea483-c5e1-4c1c-943d-e1325c0d510b" containerName="test-curl-7874-789f10"
Sep 11 17:58:05.945491 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:05.945471    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7874-bae2f4"
Sep 11 17:58:05.955216 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:05.955189    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7874-bae2f4"]
Sep 11 17:58:06.093820 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:06.093717    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q75nj\" (UniqueName: \"kubernetes.io/projected/fca94d8a-89d6-4146-8a5e-88bf1569104c-kube-api-access-q75nj\") pod \"test-curl-7874-bae2f4\" (UID: \"fca94d8a-89d6-4146-8a5e-88bf1569104c\") " pod="opendatahub/test-curl-7874-bae2f4"
Sep 11 17:58:06.194912 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:06.194874    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q75nj\" (UniqueName: \"kubernetes.io/projected/fca94d8a-89d6-4146-8a5e-88bf1569104c-kube-api-access-q75nj\") pod \"test-curl-7874-bae2f4\" (UID: \"fca94d8a-89d6-4146-8a5e-88bf1569104c\") " pod="opendatahub/test-curl-7874-bae2f4"
Sep 11 17:58:06.205124 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:06.205087    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q75nj\" (UniqueName: \"kubernetes.io/projected/fca94d8a-89d6-4146-8a5e-88bf1569104c-kube-api-access-q75nj\") pod \"test-curl-7874-bae2f4\" (UID: \"fca94d8a-89d6-4146-8a5e-88bf1569104c\") " pod="opendatahub/test-curl-7874-bae2f4"
Sep 11 17:58:06.255147 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:06.255107    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7874-bae2f4"
Sep 11 17:58:06.526540 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:06.526511    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7874-bae2f4"]
Sep 11 17:58:06.528585 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:58:06.528556    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfca94d8a_89d6_4146_8a5e_88bf1569104c.slice/crio-f153f79de32b46ab788837b39770fb5afa707bfd5f9392e30d35687cb4fe2870 WatchSource:0}: Error finding container f153f79de32b46ab788837b39770fb5afa707bfd5f9392e30d35687cb4fe2870: Status 404 returned error can't find the container with id f153f79de32b46ab788837b39770fb5afa707bfd5f9392e30d35687cb4fe2870
Sep 11 17:58:06.910329 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:06.910295    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7874-bae2f4" event={"ID":"fca94d8a-89d6-4146-8a5e-88bf1569104c","Type":"ContainerStarted","Data":"559883c75ee1303cfc35c457783800183cfd9193db741c97b96474d5fe2678b8"}
Sep 11 17:58:06.910789 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:06.910337    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7874-bae2f4" event={"ID":"fca94d8a-89d6-4146-8a5e-88bf1569104c","Type":"ContainerStarted","Data":"f153f79de32b46ab788837b39770fb5afa707bfd5f9392e30d35687cb4fe2870"}
Sep 11 17:58:06.910789 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:06.910337    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7874-789f10" podUID="667ea483-c5e1-4c1c-943d-e1325c0d510b" containerName="test-curl-7874-789f10" containerID="cri-o://a34bfe6a8fafe41bcad09f00859f7a45aca1bbe6b25c1c16e98a783fe9007715" gracePeriod=2
Sep 11 17:58:06.912905 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:58:06.912878    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7874-789f10\" is forbidden: User \"system:node:ip-10-0-138-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-96.ec2.internal' and this object" podUID="667ea483-c5e1-4c1c-943d-e1325c0d510b" pod="opendatahub/test-curl-7874-789f10"
Sep 11 17:58:06.963090 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:58:06.963051    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7874-789f10\" is forbidden: User \"system:node:ip-10-0-138-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-96.ec2.internal' and this object" podUID="667ea483-c5e1-4c1c-943d-e1325c0d510b" pod="opendatahub/test-curl-7874-789f10"
Sep 11 17:58:07.620324 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:07.620263    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7874-bae2f4" podStartSLOduration=2.620245456 podStartE2EDuration="2.620245456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:58:06.956548279 +0000 UTC m=+792.401648852" watchObservedRunningTime="2026-09-11 17:58:07.620245456 +0000 UTC m=+793.065346026"
Sep 11 17:58:07.620647 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:07.620615    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7874-bae2f4"]
Sep 11 17:58:07.623528 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:07.623453    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7874-bae2f4"]
Sep 11 17:58:07.915131 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:58:07.915089    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7874-bae2f4" container="test-curl-7874-bae2f4"
Sep 11 17:58:08.921664 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:08.921621    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7874-bae2f4" podUID="fca94d8a-89d6-4146-8a5e-88bf1569104c" containerName="test-curl-7874-bae2f4" containerID="cri-o://559883c75ee1303cfc35c457783800183cfd9193db741c97b96474d5fe2678b8" gracePeriod=2
Sep 11 17:58:08.924979 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:58:08.924942    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7874-bae2f4\" is forbidden: User \"system:node:ip-10-0-138-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-96.ec2.internal' and this object" podUID="fca94d8a-89d6-4146-8a5e-88bf1569104c" pod="opendatahub/test-curl-7874-bae2f4"
Sep 11 17:58:09.095376 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:09.095336    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7874-789f10"
Sep 11 17:58:09.223060 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:09.222979    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9f995\" (UniqueName: \"kubernetes.io/projected/667ea483-c5e1-4c1c-943d-e1325c0d510b-kube-api-access-9f995\") pod \"667ea483-c5e1-4c1c-943d-e1325c0d510b\" (UID: \"667ea483-c5e1-4c1c-943d-e1325c0d510b\") "
Sep 11 17:58:09.225199 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:09.225162    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667ea483-c5e1-4c1c-943d-e1325c0d510b-kube-api-access-9f995" (OuterVolumeSpecName: "kube-api-access-9f995") pod "667ea483-c5e1-4c1c-943d-e1325c0d510b" (UID: "667ea483-c5e1-4c1c-943d-e1325c0d510b"). InnerVolumeSpecName "kube-api-access-9f995". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:58:09.324466 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:09.324423    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f995\" (UniqueName: \"kubernetes.io/projected/667ea483-c5e1-4c1c-943d-e1325c0d510b-kube-api-access-9f995\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:58:09.929061 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:09.929023    2559 generic.go:356] "Generic (PLEG): container finished" podID="667ea483-c5e1-4c1c-943d-e1325c0d510b" containerID="a34bfe6a8fafe41bcad09f00859f7a45aca1bbe6b25c1c16e98a783fe9007715" exitCode=137
Sep 11 17:58:09.929566 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:09.929069    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7874-789f10"
Sep 11 17:58:09.929566 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:09.929128    2559 scope.go:117] "RemoveContainer" containerID="a34bfe6a8fafe41bcad09f00859f7a45aca1bbe6b25c1c16e98a783fe9007715"
Sep 11 17:58:09.939069 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:09.939048    2559 scope.go:117] "RemoveContainer" containerID="a34bfe6a8fafe41bcad09f00859f7a45aca1bbe6b25c1c16e98a783fe9007715"
Sep 11 17:58:09.939421 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:58:09.939402    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a34bfe6a8fafe41bcad09f00859f7a45aca1bbe6b25c1c16e98a783fe9007715\": container with ID starting with a34bfe6a8fafe41bcad09f00859f7a45aca1bbe6b25c1c16e98a783fe9007715 not found: ID does not exist" containerID="a34bfe6a8fafe41bcad09f00859f7a45aca1bbe6b25c1c16e98a783fe9007715"
Sep 11 17:58:09.939501 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:09.939430    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a34bfe6a8fafe41bcad09f00859f7a45aca1bbe6b25c1c16e98a783fe9007715"} err="failed to get container status \"a34bfe6a8fafe41bcad09f00859f7a45aca1bbe6b25c1c16e98a783fe9007715\": rpc error: code = NotFound desc = could not find container \"a34bfe6a8fafe41bcad09f00859f7a45aca1bbe6b25c1c16e98a783fe9007715\": container with ID starting with a34bfe6a8fafe41bcad09f00859f7a45aca1bbe6b25c1c16e98a783fe9007715 not found: ID does not exist"
Sep 11 17:58:09.940508 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:58:09.940486    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7874-789f10\" is forbidden: User \"system:node:ip-10-0-138-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-96.ec2.internal' and this object" podUID="667ea483-c5e1-4c1c-943d-e1325c0d510b" pod="opendatahub/test-curl-7874-789f10"
Sep 11 17:58:11.055026 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:11.054990    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="667ea483-c5e1-4c1c-943d-e1325c0d510b" path="/var/lib/kubelet/pods/667ea483-c5e1-4c1c-943d-e1325c0d510b/volumes"
Sep 11 17:58:11.102961 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:11.102933    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7874-bae2f4"
Sep 11 17:58:11.241576 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:11.241485    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q75nj\" (UniqueName: \"kubernetes.io/projected/fca94d8a-89d6-4146-8a5e-88bf1569104c-kube-api-access-q75nj\") pod \"fca94d8a-89d6-4146-8a5e-88bf1569104c\" (UID: \"fca94d8a-89d6-4146-8a5e-88bf1569104c\") "
Sep 11 17:58:11.243559 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:11.243526    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca94d8a-89d6-4146-8a5e-88bf1569104c-kube-api-access-q75nj" (OuterVolumeSpecName: "kube-api-access-q75nj") pod "fca94d8a-89d6-4146-8a5e-88bf1569104c" (UID: "fca94d8a-89d6-4146-8a5e-88bf1569104c"). InnerVolumeSpecName "kube-api-access-q75nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:58:11.342543 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:11.342505    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q75nj\" (UniqueName: \"kubernetes.io/projected/fca94d8a-89d6-4146-8a5e-88bf1569104c-kube-api-access-q75nj\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:58:11.944657 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:11.944619    2559 generic.go:356] "Generic (PLEG): container finished" podID="fca94d8a-89d6-4146-8a5e-88bf1569104c" containerID="559883c75ee1303cfc35c457783800183cfd9193db741c97b96474d5fe2678b8" exitCode=137
Sep 11 17:58:11.944863 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:11.944664    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7874-bae2f4"
Sep 11 17:58:11.944863 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:11.944678    2559 scope.go:117] "RemoveContainer" containerID="559883c75ee1303cfc35c457783800183cfd9193db741c97b96474d5fe2678b8"
Sep 11 17:58:11.954493 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:11.954476    2559 scope.go:117] "RemoveContainer" containerID="559883c75ee1303cfc35c457783800183cfd9193db741c97b96474d5fe2678b8"
Sep 11 17:58:11.954775 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:58:11.954756    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"559883c75ee1303cfc35c457783800183cfd9193db741c97b96474d5fe2678b8\": container with ID starting with 559883c75ee1303cfc35c457783800183cfd9193db741c97b96474d5fe2678b8 not found: ID does not exist" containerID="559883c75ee1303cfc35c457783800183cfd9193db741c97b96474d5fe2678b8"
Sep 11 17:58:11.954818 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:11.954785    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559883c75ee1303cfc35c457783800183cfd9193db741c97b96474d5fe2678b8"} err="failed to get container status \"559883c75ee1303cfc35c457783800183cfd9193db741c97b96474d5fe2678b8\": rpc error: code = NotFound desc = could not find container \"559883c75ee1303cfc35c457783800183cfd9193db741c97b96474d5fe2678b8\": container with ID starting with 559883c75ee1303cfc35c457783800183cfd9193db741c97b96474d5fe2678b8 not found: ID does not exist"
Sep 11 17:58:11.957474 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:58:11.957455    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7874-bae2f4\" is forbidden: User \"system:node:ip-10-0-138-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-96.ec2.internal' and this object" podUID="fca94d8a-89d6-4146-8a5e-88bf1569104c" pod="opendatahub/test-curl-7874-bae2f4"
Sep 11 17:58:13.057023 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.056983    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fca94d8a-89d6-4146-8a5e-88bf1569104c" path="/var/lib/kubelet/pods/fca94d8a-89d6-4146-8a5e-88bf1569104c/volumes"
Sep 11 17:58:13.087518 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.087484    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"]
Sep 11 17:58:13.087843 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.087830    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fca94d8a-89d6-4146-8a5e-88bf1569104c" containerName="test-curl-7874-bae2f4"
Sep 11 17:58:13.087888 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.087844    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fca94d8a-89d6-4146-8a5e-88bf1569104c" containerName="test-curl-7874-bae2f4"
Sep 11 17:58:13.087919 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.087904    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fca94d8a-89d6-4146-8a5e-88bf1569104c" containerName="test-curl-7874-bae2f4"
Sep 11 17:58:13.092262 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.092242    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"
Sep 11 17:58:13.104584 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.104556    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-5883b7-serving-cert\""
Sep 11 17:58:13.104724 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.104580    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:58:13.105465 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.105449    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 17:58:13.105562 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.105509    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 17:58:13.105562 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.105527    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cdcpv\""
Sep 11 17:58:13.137554 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.137520    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"]
Sep 11 17:58:13.160152 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.160116    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-metrics-certs\") pod \"maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49\" (UID: \"9595f4ff-9001-424f-ba1e-b8fe8af1fa92\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"
Sep 11 17:58:13.160385 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.160156    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmrsc\" (UniqueName: \"kubernetes.io/projected/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-kube-api-access-wmrsc\") pod \"maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49\" (UID: \"9595f4ff-9001-424f-ba1e-b8fe8af1fa92\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"
Sep 11 17:58:13.160385 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.160209    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-maas-api-tls\") pod \"maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49\" (UID: \"9595f4ff-9001-424f-ba1e-b8fe8af1fa92\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"
Sep 11 17:58:13.160385 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.160328    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-openshift-service-ca\") pod \"maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49\" (UID: \"9595f4ff-9001-424f-ba1e-b8fe8af1fa92\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"
Sep 11 17:58:13.261660 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.261620    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-openshift-service-ca\") pod \"maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49\" (UID: \"9595f4ff-9001-424f-ba1e-b8fe8af1fa92\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"
Sep 11 17:58:13.261836 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.261689    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-metrics-certs\") pod \"maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49\" (UID: \"9595f4ff-9001-424f-ba1e-b8fe8af1fa92\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"
Sep 11 17:58:13.261836 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.261717    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmrsc\" (UniqueName: \"kubernetes.io/projected/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-kube-api-access-wmrsc\") pod \"maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49\" (UID: \"9595f4ff-9001-424f-ba1e-b8fe8af1fa92\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"
Sep 11 17:58:13.261836 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.261756    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-maas-api-tls\") pod \"maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49\" (UID: \"9595f4ff-9001-424f-ba1e-b8fe8af1fa92\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"
Sep 11 17:58:13.261995 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:58:13.261934    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-serving-cert: secret "maas-api-e2e-autores-5883b7-serving-cert" not found
Sep 11 17:58:13.262042 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:58:13.262005    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-maas-api-tls podName:9595f4ff-9001-424f-ba1e-b8fe8af1fa92 nodeName:}" failed. No retries permitted until 2026-09-11 17:58:13.76198159 +0000 UTC m=+799.207082149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-maas-api-tls") pod "maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49" (UID: "9595f4ff-9001-424f-ba1e-b8fe8af1fa92") : secret "maas-api-e2e-autores-5883b7-serving-cert" not found
Sep 11 17:58:13.262417 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.262394    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-openshift-service-ca\") pod \"maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49\" (UID: \"9595f4ff-9001-424f-ba1e-b8fe8af1fa92\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"
Sep 11 17:58:13.264365 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.264324    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-metrics-certs\") pod \"maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49\" (UID: \"9595f4ff-9001-424f-ba1e-b8fe8af1fa92\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"
Sep 11 17:58:13.277732 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.277695    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmrsc\" (UniqueName: \"kubernetes.io/projected/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-kube-api-access-wmrsc\") pod \"maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49\" (UID: \"9595f4ff-9001-424f-ba1e-b8fe8af1fa92\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"
Sep 11 17:58:13.326081 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.325980    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 17:58:13.766628 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.766586    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-maas-api-tls\") pod \"maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49\" (UID: \"9595f4ff-9001-424f-ba1e-b8fe8af1fa92\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"
Sep 11 17:58:13.769105 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:13.769078    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-maas-api-tls\") pod \"maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49\" (UID: \"9595f4ff-9001-424f-ba1e-b8fe8af1fa92\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"
Sep 11 17:58:14.003411 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:14.003339    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"
Sep 11 17:58:14.313092 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:14.313063    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"]
Sep 11 17:58:14.314767 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:58:14.314733    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9595f4ff_9001_424f_ba1e_b8fe8af1fa92.slice/crio-36c82164bcbec0ab5132c667c7c0aa240ae50447464135d1369c13dc652ff491 WatchSource:0}: Error finding container 36c82164bcbec0ab5132c667c7c0aa240ae50447464135d1369c13dc652ff491: Status 404 returned error can't find the container with id 36c82164bcbec0ab5132c667c7c0aa240ae50447464135d1369c13dc652ff491
Sep 11 17:58:14.909634 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:14.909594    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"]
Sep 11 17:58:14.914094 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:14.914064    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"
Sep 11 17:58:14.918202 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:14.918175    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f412cf1d-serving-cert\""
Sep 11 17:58:14.966477 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:14.966409    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49" event={"ID":"9595f4ff-9001-424f-ba1e-b8fe8af1fa92","Type":"ContainerStarted","Data":"36c82164bcbec0ab5132c667c7c0aa240ae50447464135d1369c13dc652ff491"}
Sep 11 17:58:14.967850 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:14.967815    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"]
Sep 11 17:58:15.080269 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:15.080209    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j84rg\" (UniqueName: \"kubernetes.io/projected/55d70eec-dfb8-4bd1-8b98-632789086963-kube-api-access-j84rg\") pod \"maas-api-e2e-ait-f412cf1d-94f547798-mwpbf\" (UID: \"55d70eec-dfb8-4bd1-8b98-632789086963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"
Sep 11 17:58:15.080492 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:15.080325    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55d70eec-dfb8-4bd1-8b98-632789086963-maas-api-tls\") pod \"maas-api-e2e-ait-f412cf1d-94f547798-mwpbf\" (UID: \"55d70eec-dfb8-4bd1-8b98-632789086963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"
Sep 11 17:58:15.080492 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:15.080410    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55d70eec-dfb8-4bd1-8b98-632789086963-metrics-certs\") pod \"maas-api-e2e-ait-f412cf1d-94f547798-mwpbf\" (UID: \"55d70eec-dfb8-4bd1-8b98-632789086963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"
Sep 11 17:58:15.080607 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:15.080505    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d70eec-dfb8-4bd1-8b98-632789086963-openshift-service-ca\") pod \"maas-api-e2e-ait-f412cf1d-94f547798-mwpbf\" (UID: \"55d70eec-dfb8-4bd1-8b98-632789086963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"
Sep 11 17:58:15.181697 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:15.181601    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j84rg\" (UniqueName: \"kubernetes.io/projected/55d70eec-dfb8-4bd1-8b98-632789086963-kube-api-access-j84rg\") pod \"maas-api-e2e-ait-f412cf1d-94f547798-mwpbf\" (UID: \"55d70eec-dfb8-4bd1-8b98-632789086963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"
Sep 11 17:58:15.181697 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:15.181663    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55d70eec-dfb8-4bd1-8b98-632789086963-maas-api-tls\") pod \"maas-api-e2e-ait-f412cf1d-94f547798-mwpbf\" (UID: \"55d70eec-dfb8-4bd1-8b98-632789086963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"
Sep 11 17:58:15.181921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:15.181709    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55d70eec-dfb8-4bd1-8b98-632789086963-metrics-certs\") pod \"maas-api-e2e-ait-f412cf1d-94f547798-mwpbf\" (UID: \"55d70eec-dfb8-4bd1-8b98-632789086963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"
Sep 11 17:58:15.181921 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:15.181749    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d70eec-dfb8-4bd1-8b98-632789086963-openshift-service-ca\") pod \"maas-api-e2e-ait-f412cf1d-94f547798-mwpbf\" (UID: \"55d70eec-dfb8-4bd1-8b98-632789086963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"
Sep 11 17:58:15.182644 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:58:15.182136    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-serving-cert: secret "maas-api-e2e-ait-f412cf1d-serving-cert" not found
Sep 11 17:58:15.182644 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:58:15.182212    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55d70eec-dfb8-4bd1-8b98-632789086963-maas-api-tls podName:55d70eec-dfb8-4bd1-8b98-632789086963 nodeName:}" failed. No retries permitted until 2026-09-11 17:58:15.68218949 +0000 UTC m=+801.127290056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/55d70eec-dfb8-4bd1-8b98-632789086963-maas-api-tls") pod "maas-api-e2e-ait-f412cf1d-94f547798-mwpbf" (UID: "55d70eec-dfb8-4bd1-8b98-632789086963") : secret "maas-api-e2e-ait-f412cf1d-serving-cert" not found
Sep 11 17:58:15.182890 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:15.182646    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d70eec-dfb8-4bd1-8b98-632789086963-openshift-service-ca\") pod \"maas-api-e2e-ait-f412cf1d-94f547798-mwpbf\" (UID: \"55d70eec-dfb8-4bd1-8b98-632789086963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"
Sep 11 17:58:15.194314 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:15.185323    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55d70eec-dfb8-4bd1-8b98-632789086963-metrics-certs\") pod \"maas-api-e2e-ait-f412cf1d-94f547798-mwpbf\" (UID: \"55d70eec-dfb8-4bd1-8b98-632789086963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"
Sep 11 17:58:15.239443 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:15.239382    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j84rg\" (UniqueName: \"kubernetes.io/projected/55d70eec-dfb8-4bd1-8b98-632789086963-kube-api-access-j84rg\") pod \"maas-api-e2e-ait-f412cf1d-94f547798-mwpbf\" (UID: \"55d70eec-dfb8-4bd1-8b98-632789086963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"
Sep 11 17:58:15.686996 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:15.686956    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55d70eec-dfb8-4bd1-8b98-632789086963-maas-api-tls\") pod \"maas-api-e2e-ait-f412cf1d-94f547798-mwpbf\" (UID: \"55d70eec-dfb8-4bd1-8b98-632789086963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"
Sep 11 17:58:15.689802 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:15.689773    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55d70eec-dfb8-4bd1-8b98-632789086963-maas-api-tls\") pod \"maas-api-e2e-ait-f412cf1d-94f547798-mwpbf\" (UID: \"55d70eec-dfb8-4bd1-8b98-632789086963\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"
Sep 11 17:58:15.829706 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:15.829665    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"
Sep 11 17:58:16.847177 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:16.847091    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"]
Sep 11 17:58:16.849657 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:58:16.849624    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d70eec_dfb8_4bd1_8b98_632789086963.slice/crio-dac29872a48e9dfe6b15d7bbe21240b9b98fca80774c733328da3bf5d4339e17 WatchSource:0}: Error finding container dac29872a48e9dfe6b15d7bbe21240b9b98fca80774c733328da3bf5d4339e17: Status 404 returned error can't find the container with id dac29872a48e9dfe6b15d7bbe21240b9b98fca80774c733328da3bf5d4339e17
Sep 11 17:58:16.979512 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:16.979472    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf" event={"ID":"55d70eec-dfb8-4bd1-8b98-632789086963","Type":"ContainerStarted","Data":"dac29872a48e9dfe6b15d7bbe21240b9b98fca80774c733328da3bf5d4339e17"}
Sep 11 17:58:16.981028 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:16.981002    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49" event={"ID":"9595f4ff-9001-424f-ba1e-b8fe8af1fa92","Type":"ContainerStarted","Data":"a59b2a4c95bda7bcbe7f205f0aea632901cd6dfb0f7b7acb6716478462d82e29"}
Sep 11 17:58:16.981157 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:16.981106    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"
Sep 11 17:58:17.030984 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:17.030926    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49" podStartSLOduration=1.732470127 podStartE2EDuration="4.030909012s" totalImagesPullingTime="2.298438885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:58:17.027518865 +0000 UTC m=+802.472619436" watchObservedRunningTime="2026-09-11 17:58:17.030909012 +0000 UTC m=+802.476009583"
Sep 11 17:58:17.989174 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:17.989128    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf" event={"ID":"55d70eec-dfb8-4bd1-8b98-632789086963","Type":"ContainerStarted","Data":"23e103654dca112c78620717bb4de02c7a8d2387e8f6e3d915dd2437a40bf020"}
Sep 11 17:58:17.989755 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:17.989702    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"
Sep 11 17:58:18.027883 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:18.027816    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf" podStartSLOduration=3.699717889 podStartE2EDuration="4.027795403s" totalImagesPullingTime="328.077514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:58:18.024584336 +0000 UTC m=+803.469684911" watchObservedRunningTime="2026-09-11 17:58:18.027795403 +0000 UTC m=+803.472895975"
Sep 11 17:58:22.994823 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:22.994794    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"
Sep 11 17:58:24.080634 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:24.080597    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 17:58:25.006173 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:25.006144    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"
Sep 11 17:58:27.537076 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:27.537034    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"]
Sep 11 17:58:27.540775 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:27.540754    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"
Sep 11 17:58:27.543405 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:27.543380    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-fb8c5d-serving-cert\""
Sep 11 17:58:27.550806 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:27.550778    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"]
Sep 11 17:58:27.597986 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:27.597949    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b217c2-e087-40d0-878c-1224cd1675e7-maas-api-tls\") pod \"maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5\" (UID: \"b8b217c2-e087-40d0-878c-1224cd1675e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"
Sep 11 17:58:27.598157 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:27.598052    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn4c9\" (UniqueName: \"kubernetes.io/projected/b8b217c2-e087-40d0-878c-1224cd1675e7-kube-api-access-xn4c9\") pod \"maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5\" (UID: \"b8b217c2-e087-40d0-878c-1224cd1675e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"
Sep 11 17:58:27.598223 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:27.598201    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8b217c2-e087-40d0-878c-1224cd1675e7-metrics-certs\") pod \"maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5\" (UID: \"b8b217c2-e087-40d0-878c-1224cd1675e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"
Sep 11 17:58:27.598278 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:27.598258    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b217c2-e087-40d0-878c-1224cd1675e7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5\" (UID: \"b8b217c2-e087-40d0-878c-1224cd1675e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"
Sep 11 17:58:27.698946 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:27.698900    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn4c9\" (UniqueName: \"kubernetes.io/projected/b8b217c2-e087-40d0-878c-1224cd1675e7-kube-api-access-xn4c9\") pod \"maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5\" (UID: \"b8b217c2-e087-40d0-878c-1224cd1675e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"
Sep 11 17:58:27.699124 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:27.699008    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8b217c2-e087-40d0-878c-1224cd1675e7-metrics-certs\") pod \"maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5\" (UID: \"b8b217c2-e087-40d0-878c-1224cd1675e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"
Sep 11 17:58:27.699124 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:27.699048    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b217c2-e087-40d0-878c-1224cd1675e7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5\" (UID: \"b8b217c2-e087-40d0-878c-1224cd1675e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"
Sep 11 17:58:27.699277 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:27.699177    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b217c2-e087-40d0-878c-1224cd1675e7-maas-api-tls\") pod \"maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5\" (UID: \"b8b217c2-e087-40d0-878c-1224cd1675e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"
Sep 11 17:58:27.699335 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:58:27.699286    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-serving-cert: secret "maas-api-e2e-inf-b-fb8c5d-serving-cert" not found
Sep 11 17:58:27.699431 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:58:27.699370    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8b217c2-e087-40d0-878c-1224cd1675e7-maas-api-tls podName:b8b217c2-e087-40d0-878c-1224cd1675e7 nodeName:}" failed. No retries permitted until 2026-09-11 17:58:28.199329931 +0000 UTC m=+813.644430488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8b217c2-e087-40d0-878c-1224cd1675e7-maas-api-tls") pod "maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5" (UID: "b8b217c2-e087-40d0-878c-1224cd1675e7") : secret "maas-api-e2e-inf-b-fb8c5d-serving-cert" not found
Sep 11 17:58:27.699777 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:27.699755    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b217c2-e087-40d0-878c-1224cd1675e7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5\" (UID: \"b8b217c2-e087-40d0-878c-1224cd1675e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"
Sep 11 17:58:27.701637 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:27.701617    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8b217c2-e087-40d0-878c-1224cd1675e7-metrics-certs\") pod \"maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5\" (UID: \"b8b217c2-e087-40d0-878c-1224cd1675e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"
Sep 11 17:58:27.712127 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:27.712096    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn4c9\" (UniqueName: \"kubernetes.io/projected/b8b217c2-e087-40d0-878c-1224cd1675e7-kube-api-access-xn4c9\") pod \"maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5\" (UID: \"b8b217c2-e087-40d0-878c-1224cd1675e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"
Sep 11 17:58:28.202054 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:28.202018    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b217c2-e087-40d0-878c-1224cd1675e7-maas-api-tls\") pod \"maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5\" (UID: \"b8b217c2-e087-40d0-878c-1224cd1675e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"
Sep 11 17:58:28.202252 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:58:28.202188    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-serving-cert: secret "maas-api-e2e-inf-b-fb8c5d-serving-cert" not found
Sep 11 17:58:28.202299 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:58:28.202263    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8b217c2-e087-40d0-878c-1224cd1675e7-maas-api-tls podName:b8b217c2-e087-40d0-878c-1224cd1675e7 nodeName:}" failed. No retries permitted until 2026-09-11 17:58:29.202242565 +0000 UTC m=+814.647343119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8b217c2-e087-40d0-878c-1224cd1675e7-maas-api-tls") pod "maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5" (UID: "b8b217c2-e087-40d0-878c-1224cd1675e7") : secret "maas-api-e2e-inf-b-fb8c5d-serving-cert" not found
Sep 11 17:58:29.210166 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:29.210123    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b217c2-e087-40d0-878c-1224cd1675e7-maas-api-tls\") pod \"maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5\" (UID: \"b8b217c2-e087-40d0-878c-1224cd1675e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"
Sep 11 17:58:29.212635 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:29.212608    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b217c2-e087-40d0-878c-1224cd1675e7-maas-api-tls\") pod \"maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5\" (UID: \"b8b217c2-e087-40d0-878c-1224cd1675e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"
Sep 11 17:58:29.352623 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:29.352575    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"
Sep 11 17:58:29.706173 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:29.706140    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"]
Sep 11 17:58:29.707304 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:58:29.707278    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b217c2_e087_40d0_878c_1224cd1675e7.slice/crio-d6ea9041d06e335a172f00fc5a41bb48821adf6efb2be22baca47bbd927115c1 WatchSource:0}: Error finding container d6ea9041d06e335a172f00fc5a41bb48821adf6efb2be22baca47bbd927115c1: Status 404 returned error can't find the container with id d6ea9041d06e335a172f00fc5a41bb48821adf6efb2be22baca47bbd927115c1
Sep 11 17:58:30.053336 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:30.053305    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5" event={"ID":"b8b217c2-e087-40d0-878c-1224cd1675e7","Type":"ContainerStarted","Data":"d6ea9041d06e335a172f00fc5a41bb48821adf6efb2be22baca47bbd927115c1"}
Sep 11 17:58:31.060130 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:31.060092    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5" event={"ID":"b8b217c2-e087-40d0-878c-1224cd1675e7","Type":"ContainerStarted","Data":"f9fdd70f8ba5e332a3259d10befe1a258657c018db5ec46d63fcd1aee5bf2cfb"}
Sep 11 17:58:31.060569 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:31.060160    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"
Sep 11 17:58:31.081658 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:31.081603    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5" podStartSLOduration=3.740113042 podStartE2EDuration="4.081583222s" totalImagesPullingTime="341.47018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:58:31.080220471 +0000 UTC m=+816.525321042" watchObservedRunningTime="2026-09-11 17:58:31.081583222 +0000 UTC m=+816.526683793"
Sep 11 17:58:37.070708 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:37.070677    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"
Sep 11 17:58:40.290132 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:40.290094    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 17:58:42.054374 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:42.054327    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"]
Sep 11 17:58:42.054802 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:42.054529    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf" podUID="55d70eec-dfb8-4bd1-8b98-632789086963" containerName="maas-api" containerID="cri-o://23e103654dca112c78620717bb4de02c7a8d2387e8f6e3d915dd2437a40bf020" gracePeriod=30
Sep 11 17:58:42.314866 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:42.314801    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"
Sep 11 17:58:42.430931 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:42.430897    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d70eec-dfb8-4bd1-8b98-632789086963-openshift-service-ca\") pod \"55d70eec-dfb8-4bd1-8b98-632789086963\" (UID: \"55d70eec-dfb8-4bd1-8b98-632789086963\") "
Sep 11 17:58:42.431129 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:42.430939    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j84rg\" (UniqueName: \"kubernetes.io/projected/55d70eec-dfb8-4bd1-8b98-632789086963-kube-api-access-j84rg\") pod \"55d70eec-dfb8-4bd1-8b98-632789086963\" (UID: \"55d70eec-dfb8-4bd1-8b98-632789086963\") "
Sep 11 17:58:42.431129 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:42.430960    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55d70eec-dfb8-4bd1-8b98-632789086963-metrics-certs\") pod \"55d70eec-dfb8-4bd1-8b98-632789086963\" (UID: \"55d70eec-dfb8-4bd1-8b98-632789086963\") "
Sep 11 17:58:42.431129 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:42.430993    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55d70eec-dfb8-4bd1-8b98-632789086963-maas-api-tls\") pod \"55d70eec-dfb8-4bd1-8b98-632789086963\" (UID: \"55d70eec-dfb8-4bd1-8b98-632789086963\") "
Sep 11 17:58:42.431384 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:42.431337    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d70eec-dfb8-4bd1-8b98-632789086963-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "55d70eec-dfb8-4bd1-8b98-632789086963" (UID: "55d70eec-dfb8-4bd1-8b98-632789086963"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:58:42.433408 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:42.433381    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55d70eec-dfb8-4bd1-8b98-632789086963-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "55d70eec-dfb8-4bd1-8b98-632789086963" (UID: "55d70eec-dfb8-4bd1-8b98-632789086963"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:58:42.433509 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:42.433449    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55d70eec-dfb8-4bd1-8b98-632789086963-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "55d70eec-dfb8-4bd1-8b98-632789086963" (UID: "55d70eec-dfb8-4bd1-8b98-632789086963"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:58:42.433509 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:42.433479    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d70eec-dfb8-4bd1-8b98-632789086963-kube-api-access-j84rg" (OuterVolumeSpecName: "kube-api-access-j84rg") pod "55d70eec-dfb8-4bd1-8b98-632789086963" (UID: "55d70eec-dfb8-4bd1-8b98-632789086963"). InnerVolumeSpecName "kube-api-access-j84rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:58:42.532669 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:42.532630    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d70eec-dfb8-4bd1-8b98-632789086963-openshift-service-ca\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:58:42.533045 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:42.533013    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j84rg\" (UniqueName: \"kubernetes.io/projected/55d70eec-dfb8-4bd1-8b98-632789086963-kube-api-access-j84rg\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:58:42.533045 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:42.533043    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55d70eec-dfb8-4bd1-8b98-632789086963-metrics-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:58:42.533210 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:42.533056    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55d70eec-dfb8-4bd1-8b98-632789086963-maas-api-tls\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:58:43.130281 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:43.130245    2559 generic.go:356] "Generic (PLEG): container finished" podID="55d70eec-dfb8-4bd1-8b98-632789086963" containerID="23e103654dca112c78620717bb4de02c7a8d2387e8f6e3d915dd2437a40bf020" exitCode=0
Sep 11 17:58:43.130774 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:43.130299    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"
Sep 11 17:58:43.130774 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:43.130328    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf" event={"ID":"55d70eec-dfb8-4bd1-8b98-632789086963","Type":"ContainerDied","Data":"23e103654dca112c78620717bb4de02c7a8d2387e8f6e3d915dd2437a40bf020"}
Sep 11 17:58:43.130774 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:43.130408    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf" event={"ID":"55d70eec-dfb8-4bd1-8b98-632789086963","Type":"ContainerDied","Data":"dac29872a48e9dfe6b15d7bbe21240b9b98fca80774c733328da3bf5d4339e17"}
Sep 11 17:58:43.130774 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:43.130438    2559 scope.go:117] "RemoveContainer" containerID="23e103654dca112c78620717bb4de02c7a8d2387e8f6e3d915dd2437a40bf020"
Sep 11 17:58:43.140821 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:43.140799    2559 scope.go:117] "RemoveContainer" containerID="23e103654dca112c78620717bb4de02c7a8d2387e8f6e3d915dd2437a40bf020"
Sep 11 17:58:43.141137 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:58:43.141114    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23e103654dca112c78620717bb4de02c7a8d2387e8f6e3d915dd2437a40bf020\": container with ID starting with 23e103654dca112c78620717bb4de02c7a8d2387e8f6e3d915dd2437a40bf020 not found: ID does not exist" containerID="23e103654dca112c78620717bb4de02c7a8d2387e8f6e3d915dd2437a40bf020"
Sep 11 17:58:43.141217 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:43.141145    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23e103654dca112c78620717bb4de02c7a8d2387e8f6e3d915dd2437a40bf020"} err="failed to get container status \"23e103654dca112c78620717bb4de02c7a8d2387e8f6e3d915dd2437a40bf020\": rpc error: code = NotFound desc = could not find container \"23e103654dca112c78620717bb4de02c7a8d2387e8f6e3d915dd2437a40bf020\": container with ID starting with 23e103654dca112c78620717bb4de02c7a8d2387e8f6e3d915dd2437a40bf020 not found: ID does not exist"
Sep 11 17:58:43.170906 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:43.170872    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"]
Sep 11 17:58:43.185970 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:43.185941    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f412cf1d-94f547798-mwpbf"]
Sep 11 17:58:45.053251 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:45.053219    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55d70eec-dfb8-4bd1-8b98-632789086963" path="/var/lib/kubelet/pods/55d70eec-dfb8-4bd1-8b98-632789086963/volumes"
Sep 11 17:58:48.366802 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:48.366768    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8"]
Sep 11 17:58:48.367378 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:48.367340    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55d70eec-dfb8-4bd1-8b98-632789086963" containerName="maas-api"
Sep 11 17:58:48.367445 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:48.367382    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="55d70eec-dfb8-4bd1-8b98-632789086963" containerName="maas-api"
Sep 11 17:58:48.367491 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:48.367470    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="55d70eec-dfb8-4bd1-8b98-632789086963" containerName="maas-api"
Sep 11 17:58:48.373719 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:48.373694    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8"
Sep 11 17:58:48.376650 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:48.376627    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-5d49a629-serving-cert\""
Sep 11 17:58:48.396464 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:48.396431    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8"]
Sep 11 17:58:48.491440 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:48.491394    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3-openshift-service-ca\") pod \"maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8\" (UID: \"0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8"
Sep 11 17:58:48.491618 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:48.491476    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3-metrics-certs\") pod \"maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8\" (UID: \"0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8"
Sep 11 17:58:48.491618 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:48.491588    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3-maas-api-tls\") pod \"maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8\" (UID: \"0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8"
Sep 11 17:58:48.491698 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:48.491621    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kwbx\" (UniqueName: \"kubernetes.io/projected/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3-kube-api-access-5kwbx\") pod \"maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8\" (UID: \"0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8"
Sep 11 17:58:48.592082 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:48.592044    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3-maas-api-tls\") pod \"maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8\" (UID: \"0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8"
Sep 11 17:58:48.592258 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:48.592089    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kwbx\" (UniqueName: \"kubernetes.io/projected/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3-kube-api-access-5kwbx\") pod \"maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8\" (UID: \"0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8"
Sep 11 17:58:48.592258 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:48.592137    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3-openshift-service-ca\") pod \"maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8\" (UID: \"0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8"
Sep 11 17:58:48.592258 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:48.592152    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3-metrics-certs\") pod \"maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8\" (UID: \"0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8"
Sep 11 17:58:48.593042 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:48.593004    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3-openshift-service-ca\") pod \"maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8\" (UID: \"0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8"
Sep 11 17:58:48.594516 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:48.594494    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3-metrics-certs\") pod \"maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8\" (UID: \"0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8"
Sep 11 17:58:48.594614 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:48.594561    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3-maas-api-tls\") pod \"maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8\" (UID: \"0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8"
Sep 11 17:58:48.604095 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:48.604071    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kwbx\" (UniqueName: \"kubernetes.io/projected/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3-kube-api-access-5kwbx\") pod \"maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8\" (UID: \"0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8"
Sep 11 17:58:48.684778 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:48.684739    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8"
Sep 11 17:58:48.977634 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:48.977604    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8"]
Sep 11 17:58:48.979750 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:58:48.979722    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0615a49b_5ffa_4ce4_b03a_b6e0b14c52b3.slice/crio-8aa486f1719477a19a79fddff2f3116966a3d68a4478944baa743d0067a11e87 WatchSource:0}: Error finding container 8aa486f1719477a19a79fddff2f3116966a3d68a4478944baa743d0067a11e87: Status 404 returned error can't find the container with id 8aa486f1719477a19a79fddff2f3116966a3d68a4478944baa743d0067a11e87
Sep 11 17:58:49.166793 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:49.166755    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8" event={"ID":"0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3","Type":"ContainerStarted","Data":"8aa486f1719477a19a79fddff2f3116966a3d68a4478944baa743d0067a11e87"}
Sep 11 17:58:50.174058 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:50.174017    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8" event={"ID":"0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3","Type":"ContainerStarted","Data":"247f31f708ee749773d4429884f6761087797950de016032ca9cdf8e541fe2ed"}
Sep 11 17:58:50.174495 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:50.174184    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8"
Sep 11 17:58:50.205080 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:50.205032    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8" podStartSLOduration=1.70189218 podStartE2EDuration="2.205018021s" totalImagesPullingTime="503.125841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:58:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:58:50.202948557 +0000 UTC m=+835.648049127" watchObservedRunningTime="2026-09-11 17:58:50.205018021 +0000 UTC m=+835.650118617"
Sep 11 17:58:56.185428 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:58:56.185398    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8"
Sep 11 17:59:03.809970 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:03.809933    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 17:59:14.975058 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:14.975017    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"]
Sep 11 17:59:14.979999 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:14.979978    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 17:59:14.984731 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:14.984709    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-87ec3d46-kserve-self-signed-certs\""
Sep 11 17:59:14.998650 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:14.998614    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"]
Sep 11 17:59:15.038537 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:15.038502    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dae27d9-cc71-453d-8048-b9c350d3e95b-tmp-dir\") pod \"e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2\" (UID: \"9dae27d9-cc71-453d-8048-b9c350d3e95b\") " pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 17:59:15.038728 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:15.038550    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dae27d9-cc71-453d-8048-b9c350d3e95b-model-cache\") pod \"e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2\" (UID: \"9dae27d9-cc71-453d-8048-b9c350d3e95b\") " pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 17:59:15.038728 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:15.038580    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p2w7\" (UniqueName: \"kubernetes.io/projected/9dae27d9-cc71-453d-8048-b9c350d3e95b-kube-api-access-2p2w7\") pod \"e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2\" (UID: \"9dae27d9-cc71-453d-8048-b9c350d3e95b\") " pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 17:59:15.038728 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:15.038633    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dae27d9-cc71-453d-8048-b9c350d3e95b-tls-certs\") pod \"e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2\" (UID: \"9dae27d9-cc71-453d-8048-b9c350d3e95b\") " pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 17:59:15.038728 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:15.038711    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dae27d9-cc71-453d-8048-b9c350d3e95b-home\") pod \"e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2\" (UID: \"9dae27d9-cc71-453d-8048-b9c350d3e95b\") " pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 17:59:15.038918 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:15.038757    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dae27d9-cc71-453d-8048-b9c350d3e95b-dshm\") pod \"e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2\" (UID: \"9dae27d9-cc71-453d-8048-b9c350d3e95b\") " pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 17:59:15.139710 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:15.139665    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dae27d9-cc71-453d-8048-b9c350d3e95b-tmp-dir\") pod \"e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2\" (UID: \"9dae27d9-cc71-453d-8048-b9c350d3e95b\") " pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 17:59:15.139710 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:15.139716    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dae27d9-cc71-453d-8048-b9c350d3e95b-model-cache\") pod \"e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2\" (UID: \"9dae27d9-cc71-453d-8048-b9c350d3e95b\") " pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 17:59:15.139967 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:15.139833    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p2w7\" (UniqueName: \"kubernetes.io/projected/9dae27d9-cc71-453d-8048-b9c350d3e95b-kube-api-access-2p2w7\") pod \"e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2\" (UID: \"9dae27d9-cc71-453d-8048-b9c350d3e95b\") " pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 17:59:15.140035 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:15.139991    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dae27d9-cc71-453d-8048-b9c350d3e95b-tls-certs\") pod \"e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2\" (UID: \"9dae27d9-cc71-453d-8048-b9c350d3e95b\") " pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 17:59:15.140094 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:15.140037    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dae27d9-cc71-453d-8048-b9c350d3e95b-home\") pod \"e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2\" (UID: \"9dae27d9-cc71-453d-8048-b9c350d3e95b\") " pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 17:59:15.140094 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:15.140065    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dae27d9-cc71-453d-8048-b9c350d3e95b-dshm\") pod \"e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2\" (UID: \"9dae27d9-cc71-453d-8048-b9c350d3e95b\") " pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 17:59:15.140185 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:15.140116    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dae27d9-cc71-453d-8048-b9c350d3e95b-tmp-dir\") pod \"e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2\" (UID: \"9dae27d9-cc71-453d-8048-b9c350d3e95b\") " pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 17:59:15.140185 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:15.140131    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dae27d9-cc71-453d-8048-b9c350d3e95b-model-cache\") pod \"e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2\" (UID: \"9dae27d9-cc71-453d-8048-b9c350d3e95b\") " pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 17:59:15.140331 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:15.140310    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dae27d9-cc71-453d-8048-b9c350d3e95b-home\") pod \"e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2\" (UID: \"9dae27d9-cc71-453d-8048-b9c350d3e95b\") " pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 17:59:15.142199 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:15.142176    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dae27d9-cc71-453d-8048-b9c350d3e95b-dshm\") pod \"e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2\" (UID: \"9dae27d9-cc71-453d-8048-b9c350d3e95b\") " pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 17:59:15.142466 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:15.142447    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dae27d9-cc71-453d-8048-b9c350d3e95b-tls-certs\") pod \"e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2\" (UID: \"9dae27d9-cc71-453d-8048-b9c350d3e95b\") " pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 17:59:15.157066 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:15.157039    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p2w7\" (UniqueName: \"kubernetes.io/projected/9dae27d9-cc71-453d-8048-b9c350d3e95b-kube-api-access-2p2w7\") pod \"e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2\" (UID: \"9dae27d9-cc71-453d-8048-b9c350d3e95b\") " pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 17:59:15.289345 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:15.289251    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 17:59:15.566771 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:15.566675    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"]
Sep 11 17:59:15.569176 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:59:15.569150    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9dae27d9_cc71_453d_8048_b9c350d3e95b.slice/crio-4fe6f572613adc73dad8a2cd358dede5e53365c052a7f8c547d02eec5dd8d276 WatchSource:0}: Error finding container 4fe6f572613adc73dad8a2cd358dede5e53365c052a7f8c547d02eec5dd8d276: Status 404 returned error can't find the container with id 4fe6f572613adc73dad8a2cd358dede5e53365c052a7f8c547d02eec5dd8d276
Sep 11 17:59:16.315677 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:16.315637    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2" event={"ID":"9dae27d9-cc71-453d-8048-b9c350d3e95b","Type":"ContainerStarted","Data":"ac7efa550dc8a9d5e6e716f4d0b37b24675a9c63e96733f52af444d8f9bff908"}
Sep 11 17:59:16.316114 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:16.315685    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2" event={"ID":"9dae27d9-cc71-453d-8048-b9c350d3e95b","Type":"ContainerStarted","Data":"4fe6f572613adc73dad8a2cd358dede5e53365c052a7f8c547d02eec5dd8d276"}
Sep 11 17:59:16.353175 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:16.353117    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2" podStartSLOduration=2.353101549 podStartE2EDuration="2.353101549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:59:16.352381144 +0000 UTC m=+861.797481716" watchObservedRunningTime="2026-09-11 17:59:16.353101549 +0000 UTC m=+861.798202120"
Sep 11 17:59:17.769814 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:17.769782    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8"]
Sep 11 17:59:17.770273 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:17.770045    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8" podUID="0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3" containerName="maas-api" containerID="cri-o://247f31f708ee749773d4429884f6761087797950de016032ca9cdf8e541fe2ed" gracePeriod=30
Sep 11 17:59:18.024202 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:18.024141    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8"
Sep 11 17:59:18.064696 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:18.064664    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3-openshift-service-ca\") pod \"0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3\" (UID: \"0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3\") "
Sep 11 17:59:18.064696 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:18.064700    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kwbx\" (UniqueName: \"kubernetes.io/projected/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3-kube-api-access-5kwbx\") pod \"0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3\" (UID: \"0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3\") "
Sep 11 17:59:18.064934 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:18.064731    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3-metrics-certs\") pod \"0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3\" (UID: \"0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3\") "
Sep 11 17:59:18.064994 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:18.064927    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3-maas-api-tls\") pod \"0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3\" (UID: \"0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3\") "
Sep 11 17:59:18.065076 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:18.065053    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3" (UID: "0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:59:18.065329 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:18.065305    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3-openshift-service-ca\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:59:18.067116 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:18.067084    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3" (UID: "0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:59:18.067385 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:18.067345    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3" (UID: "0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:59:18.067459 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:18.067383    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3-kube-api-access-5kwbx" (OuterVolumeSpecName: "kube-api-access-5kwbx") pod "0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3" (UID: "0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3"). InnerVolumeSpecName "kube-api-access-5kwbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:59:18.165755 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:18.165718    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kwbx\" (UniqueName: \"kubernetes.io/projected/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3-kube-api-access-5kwbx\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:59:18.165755 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:18.165753    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3-metrics-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:59:18.165755 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:18.165762    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3-maas-api-tls\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:59:18.330360 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:18.330257    2559 generic.go:356] "Generic (PLEG): container finished" podID="0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3" containerID="247f31f708ee749773d4429884f6761087797950de016032ca9cdf8e541fe2ed" exitCode=0
Sep 11 17:59:18.330360 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:18.330297    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8"
Sep 11 17:59:18.330360 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:18.330319    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8" event={"ID":"0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3","Type":"ContainerDied","Data":"247f31f708ee749773d4429884f6761087797950de016032ca9cdf8e541fe2ed"}
Sep 11 17:59:18.330591 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:18.330389    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8" event={"ID":"0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3","Type":"ContainerDied","Data":"8aa486f1719477a19a79fddff2f3116966a3d68a4478944baa743d0067a11e87"}
Sep 11 17:59:18.330591 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:18.330420    2559 scope.go:117] "RemoveContainer" containerID="247f31f708ee749773d4429884f6761087797950de016032ca9cdf8e541fe2ed"
Sep 11 17:59:18.340610 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:18.340590    2559 scope.go:117] "RemoveContainer" containerID="247f31f708ee749773d4429884f6761087797950de016032ca9cdf8e541fe2ed"
Sep 11 17:59:18.340961 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:59:18.340941    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"247f31f708ee749773d4429884f6761087797950de016032ca9cdf8e541fe2ed\": container with ID starting with 247f31f708ee749773d4429884f6761087797950de016032ca9cdf8e541fe2ed not found: ID does not exist" containerID="247f31f708ee749773d4429884f6761087797950de016032ca9cdf8e541fe2ed"
Sep 11 17:59:18.341039 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:18.340971    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"247f31f708ee749773d4429884f6761087797950de016032ca9cdf8e541fe2ed"} err="failed to get container status \"247f31f708ee749773d4429884f6761087797950de016032ca9cdf8e541fe2ed\": rpc error: code = NotFound desc = could not find container \"247f31f708ee749773d4429884f6761087797950de016032ca9cdf8e541fe2ed\": container with ID starting with 247f31f708ee749773d4429884f6761087797950de016032ca9cdf8e541fe2ed not found: ID does not exist"
Sep 11 17:59:18.356199 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:18.356172    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8"]
Sep 11 17:59:18.361179 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:18.361151    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5d49a629-574c467b74-pxnx8"]
Sep 11 17:59:19.053080 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:19.053046    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3" path="/var/lib/kubelet/pods/0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3/volumes"
Sep 11 17:59:19.575520 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:19.575477    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 17:59:24.063985 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:24.063943    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"]
Sep 11 17:59:24.064542 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:24.064520    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3" containerName="maas-api"
Sep 11 17:59:24.064614 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:24.064545    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3" containerName="maas-api"
Sep 11 17:59:24.064680 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:24.064636    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0615a49b-5ffa-4ce4-b03a-b6e0b14c52b3" containerName="maas-api"
Sep 11 17:59:24.067784 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:24.067763    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"
Sep 11 17:59:24.071420 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:24.071400    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e436a33c-serving-cert\""
Sep 11 17:59:24.083246 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:24.083218    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"]
Sep 11 17:59:24.123045 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:24.123005    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2w7j\" (UniqueName: \"kubernetes.io/projected/e556c690-81a9-45a1-94c1-01b97b3ca22e-kube-api-access-q2w7j\") pod \"maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h\" (UID: \"e556c690-81a9-45a1-94c1-01b97b3ca22e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"
Sep 11 17:59:24.123247 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:24.123053    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e556c690-81a9-45a1-94c1-01b97b3ca22e-openshift-service-ca\") pod \"maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h\" (UID: \"e556c690-81a9-45a1-94c1-01b97b3ca22e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"
Sep 11 17:59:24.123247 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:24.123092    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e556c690-81a9-45a1-94c1-01b97b3ca22e-metrics-certs\") pod \"maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h\" (UID: \"e556c690-81a9-45a1-94c1-01b97b3ca22e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"
Sep 11 17:59:24.123247 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:24.123220    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e556c690-81a9-45a1-94c1-01b97b3ca22e-maas-api-tls\") pod \"maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h\" (UID: \"e556c690-81a9-45a1-94c1-01b97b3ca22e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"
Sep 11 17:59:24.224642 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:24.224601    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e556c690-81a9-45a1-94c1-01b97b3ca22e-maas-api-tls\") pod \"maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h\" (UID: \"e556c690-81a9-45a1-94c1-01b97b3ca22e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"
Sep 11 17:59:24.224825 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:24.224704    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2w7j\" (UniqueName: \"kubernetes.io/projected/e556c690-81a9-45a1-94c1-01b97b3ca22e-kube-api-access-q2w7j\") pod \"maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h\" (UID: \"e556c690-81a9-45a1-94c1-01b97b3ca22e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"
Sep 11 17:59:24.224825 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:24.224732    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e556c690-81a9-45a1-94c1-01b97b3ca22e-openshift-service-ca\") pod \"maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h\" (UID: \"e556c690-81a9-45a1-94c1-01b97b3ca22e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"
Sep 11 17:59:24.224825 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:59:24.224751    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-serving-cert: secret "maas-api-e2e-ait-e436a33c-serving-cert" not found
Sep 11 17:59:24.224825 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:24.224769    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e556c690-81a9-45a1-94c1-01b97b3ca22e-metrics-certs\") pod \"maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h\" (UID: \"e556c690-81a9-45a1-94c1-01b97b3ca22e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"
Sep 11 17:59:24.225028 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:59:24.224827    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e556c690-81a9-45a1-94c1-01b97b3ca22e-maas-api-tls podName:e556c690-81a9-45a1-94c1-01b97b3ca22e nodeName:}" failed. No retries permitted until 2026-09-11 17:59:24.724805254 +0000 UTC m=+870.169905826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e556c690-81a9-45a1-94c1-01b97b3ca22e-maas-api-tls") pod "maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h" (UID: "e556c690-81a9-45a1-94c1-01b97b3ca22e") : secret "maas-api-e2e-ait-e436a33c-serving-cert" not found
Sep 11 17:59:24.225456 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:24.225432    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e556c690-81a9-45a1-94c1-01b97b3ca22e-openshift-service-ca\") pod \"maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h\" (UID: \"e556c690-81a9-45a1-94c1-01b97b3ca22e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"
Sep 11 17:59:24.227276 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:24.227256    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e556c690-81a9-45a1-94c1-01b97b3ca22e-metrics-certs\") pod \"maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h\" (UID: \"e556c690-81a9-45a1-94c1-01b97b3ca22e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"
Sep 11 17:59:24.235458 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:24.235433    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2w7j\" (UniqueName: \"kubernetes.io/projected/e556c690-81a9-45a1-94c1-01b97b3ca22e-kube-api-access-q2w7j\") pod \"maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h\" (UID: \"e556c690-81a9-45a1-94c1-01b97b3ca22e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"
Sep 11 17:59:24.728874 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:24.728834    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e556c690-81a9-45a1-94c1-01b97b3ca22e-maas-api-tls\") pod \"maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h\" (UID: \"e556c690-81a9-45a1-94c1-01b97b3ca22e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"
Sep 11 17:59:24.731256 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:24.731222    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e556c690-81a9-45a1-94c1-01b97b3ca22e-maas-api-tls\") pod \"maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h\" (UID: \"e556c690-81a9-45a1-94c1-01b97b3ca22e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"
Sep 11 17:59:24.978072 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:24.978034    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"
Sep 11 17:59:25.288036 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:25.287957    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"]
Sep 11 17:59:25.290131 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:25.290110    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 17:59:25.290253 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:25.290195    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 17:59:25.291479 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:59:25.291453    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode556c690_81a9_45a1_94c1_01b97b3ca22e.slice/crio-30088559ed827b7b2093f2d1efec674a89e11340db03086259f6508bc2df522d WatchSource:0}: Error finding container 30088559ed827b7b2093f2d1efec674a89e11340db03086259f6508bc2df522d: Status 404 returned error can't find the container with id 30088559ed827b7b2093f2d1efec674a89e11340db03086259f6508bc2df522d
Sep 11 17:59:25.304164 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:25.304137    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 17:59:25.374180 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:25.374145    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h" event={"ID":"e556c690-81a9-45a1-94c1-01b97b3ca22e","Type":"ContainerStarted","Data":"30088559ed827b7b2093f2d1efec674a89e11340db03086259f6508bc2df522d"}
Sep 11 17:59:25.385589 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:25.385555    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 17:59:26.382191 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:26.382143    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h" event={"ID":"e556c690-81a9-45a1-94c1-01b97b3ca22e","Type":"ContainerStarted","Data":"257df98c6074285fae4b05b296d04739b7ab6c194df2aeb91f4d11f278bd62e1"}
Sep 11 17:59:26.382650 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:26.382389    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"
Sep 11 17:59:26.415015 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:26.414950    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h" podStartSLOduration=1.9663031800000002 podStartE2EDuration="2.414932139s" totalImagesPullingTime="448.628959ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:59:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:59:26.411282671 +0000 UTC m=+871.856383242" watchObservedRunningTime="2026-09-11 17:59:26.414932139 +0000 UTC m=+871.860032746"
Sep 11 17:59:32.393842 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:32.393812    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"
Sep 11 17:59:37.756471 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.756426    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"]
Sep 11 17:59:37.761579 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.761554    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 17:59:37.765141 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.765118    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-5883b7\"/\"kube-root-ca.crt\""
Sep 11 17:59:37.765278 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.765136    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-5883b7\"/\"autores-model-5883b7-kserve-self-signed-certs\""
Sep 11 17:59:37.765449 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.765434    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-5883b7\"/\"default-dockercfg-r5zp7\""
Sep 11 17:59:37.765537 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.765483    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-5883b7\"/\"openshift-service-ca.crt\""
Sep 11 17:59:37.770448 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.770427    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"]
Sep 11 17:59:37.857480 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.857448    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-tmp-dir\") pod \"autores-model-5883b7-kserve-765d54dbd7-5szf7\" (UID: \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\") " pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 17:59:37.857638 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.857486    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-tls-certs\") pod \"autores-model-5883b7-kserve-765d54dbd7-5szf7\" (UID: \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\") " pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 17:59:37.857638 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.857628    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-model-cache\") pod \"autores-model-5883b7-kserve-765d54dbd7-5szf7\" (UID: \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\") " pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 17:59:37.857720 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.857672    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-home\") pod \"autores-model-5883b7-kserve-765d54dbd7-5szf7\" (UID: \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\") " pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 17:59:37.857720 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.857693    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-dshm\") pod \"autores-model-5883b7-kserve-765d54dbd7-5szf7\" (UID: \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\") " pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 17:59:37.857805 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.857744    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtll9\" (UniqueName: \"kubernetes.io/projected/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-kube-api-access-vtll9\") pod \"autores-model-5883b7-kserve-765d54dbd7-5szf7\" (UID: \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\") " pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 17:59:37.958712 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.958671    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-model-cache\") pod \"autores-model-5883b7-kserve-765d54dbd7-5szf7\" (UID: \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\") " pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 17:59:37.958892 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.958724    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-home\") pod \"autores-model-5883b7-kserve-765d54dbd7-5szf7\" (UID: \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\") " pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 17:59:37.958892 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.958844    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-dshm\") pod \"autores-model-5883b7-kserve-765d54dbd7-5szf7\" (UID: \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\") " pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 17:59:37.959021 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.958897    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtll9\" (UniqueName: \"kubernetes.io/projected/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-kube-api-access-vtll9\") pod \"autores-model-5883b7-kserve-765d54dbd7-5szf7\" (UID: \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\") " pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 17:59:37.959021 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.958955    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-tmp-dir\") pod \"autores-model-5883b7-kserve-765d54dbd7-5szf7\" (UID: \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\") " pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 17:59:37.959021 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.958985    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-tls-certs\") pod \"autores-model-5883b7-kserve-765d54dbd7-5szf7\" (UID: \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\") " pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 17:59:37.959178 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.959061    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-home\") pod \"autores-model-5883b7-kserve-765d54dbd7-5szf7\" (UID: \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\") " pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 17:59:37.959178 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.959107    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-model-cache\") pod \"autores-model-5883b7-kserve-765d54dbd7-5szf7\" (UID: \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\") " pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 17:59:37.959287 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.959239    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-tmp-dir\") pod \"autores-model-5883b7-kserve-765d54dbd7-5szf7\" (UID: \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\") " pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 17:59:37.961174 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.961151    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-dshm\") pod \"autores-model-5883b7-kserve-765d54dbd7-5szf7\" (UID: \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\") " pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 17:59:37.961340 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.961325    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-tls-certs\") pod \"autores-model-5883b7-kserve-765d54dbd7-5szf7\" (UID: \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\") " pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 17:59:37.967802 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:37.967779    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtll9\" (UniqueName: \"kubernetes.io/projected/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-kube-api-access-vtll9\") pod \"autores-model-5883b7-kserve-765d54dbd7-5szf7\" (UID: \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\") " pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 17:59:38.071655 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:38.071556    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 17:59:38.238266 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:38.238225    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 17:59:38.350094 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:38.350065    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"]
Sep 11 17:59:38.351441 ip-10-0-138-96 kubenswrapper[2559]: W0911 17:59:38.351409    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b8bc1d7_09f9_48c2_94be_0ae310ca2a89.slice/crio-ee82da552b2c29e3e3da1d0c5553d65673d8f7e06c9bdc73aa71f18db09b66cc WatchSource:0}: Error finding container ee82da552b2c29e3e3da1d0c5553d65673d8f7e06c9bdc73aa71f18db09b66cc: Status 404 returned error can't find the container with id ee82da552b2c29e3e3da1d0c5553d65673d8f7e06c9bdc73aa71f18db09b66cc
Sep 11 17:59:38.450537 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:38.450489    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7" event={"ID":"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89","Type":"ContainerStarted","Data":"ad4686bb0053c604bdb232520ddeb49e436c29a1b1ff8f5ceae137faba24c45c"}
Sep 11 17:59:38.450728 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:38.450541    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7" event={"ID":"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89","Type":"ContainerStarted","Data":"ee82da552b2c29e3e3da1d0c5553d65673d8f7e06c9bdc73aa71f18db09b66cc"}
Sep 11 17:59:48.071977 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:48.071892    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 17:59:48.072451 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:48.072218    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 17:59:48.085249 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:48.085221    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 17:59:48.107085 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:48.107031    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7" podStartSLOduration=11.107013742 podStartE2EDuration="11.107013742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:59:38.482319325 +0000 UTC m=+883.927419896" watchObservedRunningTime="2026-09-11 17:59:48.107013742 +0000 UTC m=+893.552114312"
Sep 11 17:59:48.527273 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:48.527242    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 17:59:53.437275 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:53.437226    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 17:59:54.571287 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:54.571252    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"]
Sep 11 17:59:54.571793 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:54.571538    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h" podUID="e556c690-81a9-45a1-94c1-01b97b3ca22e" containerName="maas-api" containerID="cri-o://257df98c6074285fae4b05b296d04739b7ab6c194df2aeb91f4d11f278bd62e1" gracePeriod=30
Sep 11 17:59:54.820198 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:54.820175    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"
Sep 11 17:59:54.929236 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:54.929197    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e556c690-81a9-45a1-94c1-01b97b3ca22e-maas-api-tls\") pod \"e556c690-81a9-45a1-94c1-01b97b3ca22e\" (UID: \"e556c690-81a9-45a1-94c1-01b97b3ca22e\") "
Sep 11 17:59:54.929451 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:54.929311    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e556c690-81a9-45a1-94c1-01b97b3ca22e-metrics-certs\") pod \"e556c690-81a9-45a1-94c1-01b97b3ca22e\" (UID: \"e556c690-81a9-45a1-94c1-01b97b3ca22e\") "
Sep 11 17:59:54.929451 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:54.929329    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2w7j\" (UniqueName: \"kubernetes.io/projected/e556c690-81a9-45a1-94c1-01b97b3ca22e-kube-api-access-q2w7j\") pod \"e556c690-81a9-45a1-94c1-01b97b3ca22e\" (UID: \"e556c690-81a9-45a1-94c1-01b97b3ca22e\") "
Sep 11 17:59:54.929451 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:54.929400    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e556c690-81a9-45a1-94c1-01b97b3ca22e-openshift-service-ca\") pod \"e556c690-81a9-45a1-94c1-01b97b3ca22e\" (UID: \"e556c690-81a9-45a1-94c1-01b97b3ca22e\") "
Sep 11 17:59:54.929749 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:54.929721    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e556c690-81a9-45a1-94c1-01b97b3ca22e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e556c690-81a9-45a1-94c1-01b97b3ca22e" (UID: "e556c690-81a9-45a1-94c1-01b97b3ca22e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:59:54.931700 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:54.931666    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e556c690-81a9-45a1-94c1-01b97b3ca22e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e556c690-81a9-45a1-94c1-01b97b3ca22e" (UID: "e556c690-81a9-45a1-94c1-01b97b3ca22e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:59:54.931700 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:54.931682    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e556c690-81a9-45a1-94c1-01b97b3ca22e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e556c690-81a9-45a1-94c1-01b97b3ca22e" (UID: "e556c690-81a9-45a1-94c1-01b97b3ca22e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:59:54.931865 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:54.931716    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e556c690-81a9-45a1-94c1-01b97b3ca22e-kube-api-access-q2w7j" (OuterVolumeSpecName: "kube-api-access-q2w7j") pod "e556c690-81a9-45a1-94c1-01b97b3ca22e" (UID: "e556c690-81a9-45a1-94c1-01b97b3ca22e"). InnerVolumeSpecName "kube-api-access-q2w7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:59:55.030169 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:55.030136    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e556c690-81a9-45a1-94c1-01b97b3ca22e-openshift-service-ca\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:59:55.030169 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:55.030171    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e556c690-81a9-45a1-94c1-01b97b3ca22e-maas-api-tls\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:59:55.030477 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:55.030182    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e556c690-81a9-45a1-94c1-01b97b3ca22e-metrics-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:59:55.030477 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:55.030192    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2w7j\" (UniqueName: \"kubernetes.io/projected/e556c690-81a9-45a1-94c1-01b97b3ca22e-kube-api-access-q2w7j\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 17:59:55.556671 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:55.556634    2559 generic.go:356] "Generic (PLEG): container finished" podID="e556c690-81a9-45a1-94c1-01b97b3ca22e" containerID="257df98c6074285fae4b05b296d04739b7ab6c194df2aeb91f4d11f278bd62e1" exitCode=0
Sep 11 17:59:55.556844 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:55.556692    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"
Sep 11 17:59:55.556844 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:55.556720    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h" event={"ID":"e556c690-81a9-45a1-94c1-01b97b3ca22e","Type":"ContainerDied","Data":"257df98c6074285fae4b05b296d04739b7ab6c194df2aeb91f4d11f278bd62e1"}
Sep 11 17:59:55.556844 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:55.556781    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h" event={"ID":"e556c690-81a9-45a1-94c1-01b97b3ca22e","Type":"ContainerDied","Data":"30088559ed827b7b2093f2d1efec674a89e11340db03086259f6508bc2df522d"}
Sep 11 17:59:55.556844 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:55.556806    2559 scope.go:117] "RemoveContainer" containerID="257df98c6074285fae4b05b296d04739b7ab6c194df2aeb91f4d11f278bd62e1"
Sep 11 17:59:55.565756 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:55.565736    2559 scope.go:117] "RemoveContainer" containerID="257df98c6074285fae4b05b296d04739b7ab6c194df2aeb91f4d11f278bd62e1"
Sep 11 17:59:55.566039 ip-10-0-138-96 kubenswrapper[2559]: E0911 17:59:55.566021    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"257df98c6074285fae4b05b296d04739b7ab6c194df2aeb91f4d11f278bd62e1\": container with ID starting with 257df98c6074285fae4b05b296d04739b7ab6c194df2aeb91f4d11f278bd62e1 not found: ID does not exist" containerID="257df98c6074285fae4b05b296d04739b7ab6c194df2aeb91f4d11f278bd62e1"
Sep 11 17:59:55.566103 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:55.566051    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"257df98c6074285fae4b05b296d04739b7ab6c194df2aeb91f4d11f278bd62e1"} err="failed to get container status \"257df98c6074285fae4b05b296d04739b7ab6c194df2aeb91f4d11f278bd62e1\": rpc error: code = NotFound desc = could not find container \"257df98c6074285fae4b05b296d04739b7ab6c194df2aeb91f4d11f278bd62e1\": container with ID starting with 257df98c6074285fae4b05b296d04739b7ab6c194df2aeb91f4d11f278bd62e1 not found: ID does not exist"
Sep 11 17:59:55.576985 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:55.576960    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"]
Sep 11 17:59:55.586863 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:55.586839    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e436a33c-7bf9c74c47-jmz4h"]
Sep 11 17:59:57.053688 ip-10-0-138-96 kubenswrapper[2559]: I0911 17:59:57.053656    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e556c690-81a9-45a1-94c1-01b97b3ca22e" path="/var/lib/kubelet/pods/e556c690-81a9-45a1-94c1-01b97b3ca22e/volumes"
Sep 11 18:00:00.267571 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:00.267538    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819160-db8gv"]
Sep 11 18:00:00.267938 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:00.267912    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e556c690-81a9-45a1-94c1-01b97b3ca22e" containerName="maas-api"
Sep 11 18:00:00.267938 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:00.267924    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e556c690-81a9-45a1-94c1-01b97b3ca22e" containerName="maas-api"
Sep 11 18:00:00.268025 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:00.267992    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e556c690-81a9-45a1-94c1-01b97b3ca22e" containerName="maas-api"
Sep 11 18:00:00.272438 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:00.272414    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819160-db8gv"
Sep 11 18:00:00.277025 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:00.277007    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4wxpj\""
Sep 11 18:00:00.281937 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:00.281918    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819160-db8gv"
Sep 11 18:00:00.333291 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:00.333258    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819160-db8gv"]
Sep 11 18:00:00.570639 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:00.570611    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819160-db8gv"]
Sep 11 18:00:00.572143 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:00:00.572118    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22904b5e_730e_4d2f_b488_1f48cb9e2052.slice/crio-b751bc1e95d490f6ae28f88cb680db6659b3872d2964a1174ad08651d3a5ac06 WatchSource:0}: Error finding container b751bc1e95d490f6ae28f88cb680db6659b3872d2964a1174ad08651d3a5ac06: Status 404 returned error can't find the container with id b751bc1e95d490f6ae28f88cb680db6659b3872d2964a1174ad08651d3a5ac06
Sep 11 18:00:00.590956 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:00.590920    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819160-db8gv" event={"ID":"22904b5e-730e-4d2f-b488-1f48cb9e2052","Type":"ContainerStarted","Data":"b751bc1e95d490f6ae28f88cb680db6659b3872d2964a1174ad08651d3a5ac06"}
Sep 11 18:00:03.610165 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:03.610132    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819160-db8gv" event={"ID":"22904b5e-730e-4d2f-b488-1f48cb9e2052","Type":"ContainerStarted","Data":"0c82f983410e84f9851c36c8c1246ce3536dee9cf9c725ed8250d6c575e2809b"}
Sep 11 18:00:03.643136 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:03.643078    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819160-db8gv" podStartSLOduration=0.978248693 podStartE2EDuration="3.643062027s" totalImagesPullingTime="2.664813334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:00:03.640606696 +0000 UTC m=+909.085707267" watchObservedRunningTime="2026-09-11 18:00:03.643062027 +0000 UTC m=+909.088162597"
Sep 11 18:00:04.048316 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.048281    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"]
Sep 11 18:00:04.052295 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.052267    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"
Sep 11 18:00:04.056227 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.056199    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-c83b88f4-serving-cert\""
Sep 11 18:00:04.063451 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.063420    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"]
Sep 11 18:00:04.118011 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.117972    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg9vh\" (UniqueName: \"kubernetes.io/projected/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-kube-api-access-hg9vh\") pod \"maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm\" (UID: \"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"
Sep 11 18:00:04.118011 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.118013    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-metrics-certs\") pod \"maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm\" (UID: \"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"
Sep 11 18:00:04.118293 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.118153    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-maas-api-tls\") pod \"maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm\" (UID: \"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"
Sep 11 18:00:04.118293 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.118223    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-openshift-service-ca\") pod \"maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm\" (UID: \"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"
Sep 11 18:00:04.219070 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.219026    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg9vh\" (UniqueName: \"kubernetes.io/projected/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-kube-api-access-hg9vh\") pod \"maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm\" (UID: \"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"
Sep 11 18:00:04.219070 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.219071    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-metrics-certs\") pod \"maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm\" (UID: \"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"
Sep 11 18:00:04.219326 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.219156    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-maas-api-tls\") pod \"maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm\" (UID: \"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"
Sep 11 18:00:04.219326 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.219193    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-openshift-service-ca\") pod \"maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm\" (UID: \"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"
Sep 11 18:00:04.219817 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:00:04.219789    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-serving-cert: secret "maas-api-e2e-derive-c83b88f4-serving-cert" not found
Sep 11 18:00:04.219939 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:00:04.219890    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-maas-api-tls podName:2e6cded0-1fda-443d-a74a-b7c5a4f21e9b nodeName:}" failed. No retries permitted until 2026-09-11 18:00:04.719867288 +0000 UTC m=+910.164967840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-maas-api-tls") pod "maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm" (UID: "2e6cded0-1fda-443d-a74a-b7c5a4f21e9b") : secret "maas-api-e2e-derive-c83b88f4-serving-cert" not found
Sep 11 18:00:04.227900 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.220511    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-openshift-service-ca\") pod \"maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm\" (UID: \"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"
Sep 11 18:00:04.228996 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.228969    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-metrics-certs\") pod \"maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm\" (UID: \"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"
Sep 11 18:00:04.232407 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.232378    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg9vh\" (UniqueName: \"kubernetes.io/projected/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-kube-api-access-hg9vh\") pod \"maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm\" (UID: \"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"
Sep 11 18:00:04.473184 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.473147    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"]
Sep 11 18:00:04.476635 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.476618    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:00:04.479710 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.479688    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-67477b\"/\"kube-root-ca.crt\""
Sep 11 18:00:04.479912 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.479898    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-67477b\"/\"openshift-service-ca.crt\""
Sep 11 18:00:04.479966 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.479922    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-67477b\"/\"test-model-67477b-kserve-self-signed-certs\""
Sep 11 18:00:04.480191 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.480176    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-67477b\"/\"default-dockercfg-8htd4\""
Sep 11 18:00:04.492442 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.492411    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"]
Sep 11 18:00:04.523842 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.523796    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b282f7-eeab-422d-b460-0c927e390749-dshm\") pod \"test-model-67477b-kserve-55ccb45c87-qmk7k\" (UID: \"28b282f7-eeab-422d-b460-0c927e390749\") " pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:00:04.524031 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.523908    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b282f7-eeab-422d-b460-0c927e390749-tls-certs\") pod \"test-model-67477b-kserve-55ccb45c87-qmk7k\" (UID: \"28b282f7-eeab-422d-b460-0c927e390749\") " pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:00:04.524031 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.523941    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28b282f7-eeab-422d-b460-0c927e390749-home\") pod \"test-model-67477b-kserve-55ccb45c87-qmk7k\" (UID: \"28b282f7-eeab-422d-b460-0c927e390749\") " pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:00:04.524157 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.524080    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b282f7-eeab-422d-b460-0c927e390749-tmp-dir\") pod \"test-model-67477b-kserve-55ccb45c87-qmk7k\" (UID: \"28b282f7-eeab-422d-b460-0c927e390749\") " pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:00:04.524157 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.524118    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l75c\" (UniqueName: \"kubernetes.io/projected/28b282f7-eeab-422d-b460-0c927e390749-kube-api-access-6l75c\") pod \"test-model-67477b-kserve-55ccb45c87-qmk7k\" (UID: \"28b282f7-eeab-422d-b460-0c927e390749\") " pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:00:04.524259 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.524187    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b282f7-eeab-422d-b460-0c927e390749-model-cache\") pod \"test-model-67477b-kserve-55ccb45c87-qmk7k\" (UID: \"28b282f7-eeab-422d-b460-0c927e390749\") " pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:00:04.617766 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.617730    2559 generic.go:356] "Generic (PLEG): container finished" podID="22904b5e-730e-4d2f-b488-1f48cb9e2052" containerID="0c82f983410e84f9851c36c8c1246ce3536dee9cf9c725ed8250d6c575e2809b" exitCode=0
Sep 11 18:00:04.618163 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.617796    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819160-db8gv" event={"ID":"22904b5e-730e-4d2f-b488-1f48cb9e2052","Type":"ContainerDied","Data":"0c82f983410e84f9851c36c8c1246ce3536dee9cf9c725ed8250d6c575e2809b"}
Sep 11 18:00:04.624779 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.624752    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b282f7-eeab-422d-b460-0c927e390749-tmp-dir\") pod \"test-model-67477b-kserve-55ccb45c87-qmk7k\" (UID: \"28b282f7-eeab-422d-b460-0c927e390749\") " pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:00:04.624904 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.624783    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l75c\" (UniqueName: \"kubernetes.io/projected/28b282f7-eeab-422d-b460-0c927e390749-kube-api-access-6l75c\") pod \"test-model-67477b-kserve-55ccb45c87-qmk7k\" (UID: \"28b282f7-eeab-422d-b460-0c927e390749\") " pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:00:04.624904 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.624807    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b282f7-eeab-422d-b460-0c927e390749-model-cache\") pod \"test-model-67477b-kserve-55ccb45c87-qmk7k\" (UID: \"28b282f7-eeab-422d-b460-0c927e390749\") " pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:00:04.624904 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.624895    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b282f7-eeab-422d-b460-0c927e390749-dshm\") pod \"test-model-67477b-kserve-55ccb45c87-qmk7k\" (UID: \"28b282f7-eeab-422d-b460-0c927e390749\") " pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:00:04.625071 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.624952    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b282f7-eeab-422d-b460-0c927e390749-tls-certs\") pod \"test-model-67477b-kserve-55ccb45c87-qmk7k\" (UID: \"28b282f7-eeab-422d-b460-0c927e390749\") " pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:00:04.625071 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.624988    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28b282f7-eeab-422d-b460-0c927e390749-home\") pod \"test-model-67477b-kserve-55ccb45c87-qmk7k\" (UID: \"28b282f7-eeab-422d-b460-0c927e390749\") " pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:00:04.625175 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.625143    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b282f7-eeab-422d-b460-0c927e390749-tmp-dir\") pod \"test-model-67477b-kserve-55ccb45c87-qmk7k\" (UID: \"28b282f7-eeab-422d-b460-0c927e390749\") " pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:00:04.625253 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.625227    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b282f7-eeab-422d-b460-0c927e390749-model-cache\") pod \"test-model-67477b-kserve-55ccb45c87-qmk7k\" (UID: \"28b282f7-eeab-422d-b460-0c927e390749\") " pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:00:04.625323 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.625306    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28b282f7-eeab-422d-b460-0c927e390749-home\") pod \"test-model-67477b-kserve-55ccb45c87-qmk7k\" (UID: \"28b282f7-eeab-422d-b460-0c927e390749\") " pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:00:04.627191 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.627171    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b282f7-eeab-422d-b460-0c927e390749-dshm\") pod \"test-model-67477b-kserve-55ccb45c87-qmk7k\" (UID: \"28b282f7-eeab-422d-b460-0c927e390749\") " pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:00:04.627398 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.627380    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b282f7-eeab-422d-b460-0c927e390749-tls-certs\") pod \"test-model-67477b-kserve-55ccb45c87-qmk7k\" (UID: \"28b282f7-eeab-422d-b460-0c927e390749\") " pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:00:04.638929 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.638901    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l75c\" (UniqueName: \"kubernetes.io/projected/28b282f7-eeab-422d-b460-0c927e390749-kube-api-access-6l75c\") pod \"test-model-67477b-kserve-55ccb45c87-qmk7k\" (UID: \"28b282f7-eeab-422d-b460-0c927e390749\") " pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:00:04.725964 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.725863    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-maas-api-tls\") pod \"maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm\" (UID: \"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"
Sep 11 18:00:04.728275 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.728254    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-maas-api-tls\") pod \"maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm\" (UID: \"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"
Sep 11 18:00:04.786410 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.786363    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:00:04.965709 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:04.965666    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"
Sep 11 18:00:05.068637 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:00:05.068597    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28b282f7_eeab_422d_b460_0c927e390749.slice/crio-39fac2d93e3ae968b347b8974108dbe2ba43a0e81391e3cdf44f43634f0a3771 WatchSource:0}: Error finding container 39fac2d93e3ae968b347b8974108dbe2ba43a0e81391e3cdf44f43634f0a3771: Status 404 returned error can't find the container with id 39fac2d93e3ae968b347b8974108dbe2ba43a0e81391e3cdf44f43634f0a3771
Sep 11 18:00:05.068872 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:05.068847    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"]
Sep 11 18:00:05.276225 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:05.276194    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"]
Sep 11 18:00:05.276807 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:00:05.276783    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6cded0_1fda_443d_a74a_b7c5a4f21e9b.slice/crio-56ce9149959d4695a4d821054bbcdbdbefbd312a7f5d8eab1adfd1a9de6b9835 WatchSource:0}: Error finding container 56ce9149959d4695a4d821054bbcdbdbefbd312a7f5d8eab1adfd1a9de6b9835: Status 404 returned error can't find the container with id 56ce9149959d4695a4d821054bbcdbdbefbd312a7f5d8eab1adfd1a9de6b9835
Sep 11 18:00:05.624689 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:05.624590    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm" event={"ID":"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b","Type":"ContainerStarted","Data":"56ce9149959d4695a4d821054bbcdbdbefbd312a7f5d8eab1adfd1a9de6b9835"}
Sep 11 18:00:05.626149 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:05.626118    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k" event={"ID":"28b282f7-eeab-422d-b460-0c927e390749","Type":"ContainerStarted","Data":"e1fffabeaae41bbafbca9bea76272474b1424cdb48f61a1d286212a76115abcb"}
Sep 11 18:00:05.626249 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:05.626159    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k" event={"ID":"28b282f7-eeab-422d-b460-0c927e390749","Type":"ContainerStarted","Data":"39fac2d93e3ae968b347b8974108dbe2ba43a0e81391e3cdf44f43634f0a3771"}
Sep 11 18:00:05.654922 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:05.654865    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k" podStartSLOduration=1.65484885 podStartE2EDuration="1.65484885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:00:05.652863929 +0000 UTC m=+911.097964501" watchObservedRunningTime="2026-09-11 18:00:05.65484885 +0000 UTC m=+911.099949420"
Sep 11 18:00:05.755918 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:05.755890    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819160-db8gv"
Sep 11 18:00:06.633413 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:06.633374    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm" event={"ID":"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b","Type":"ContainerStarted","Data":"b46de0002af367c6ff2a120f434fe3158984a7bb25be1d70828ab97582c84043"}
Sep 11 18:00:06.633909 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:06.633470    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"
Sep 11 18:00:06.634762 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:06.634742    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819160-db8gv"
Sep 11 18:00:06.634861 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:06.634780    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819160-db8gv" event={"ID":"22904b5e-730e-4d2f-b488-1f48cb9e2052","Type":"ContainerDied","Data":"b751bc1e95d490f6ae28f88cb680db6659b3872d2964a1174ad08651d3a5ac06"}
Sep 11 18:00:06.634861 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:06.634819    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b751bc1e95d490f6ae28f88cb680db6659b3872d2964a1174ad08651d3a5ac06"
Sep 11 18:00:06.659594 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:06.659541    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm" podStartSLOduration=1.493565549 podStartE2EDuration="2.659524412s" totalImagesPullingTime="1.165958863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:00:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:00:06.656953108 +0000 UTC m=+912.102053678" watchObservedRunningTime="2026-09-11 18:00:06.659524412 +0000 UTC m=+912.104624973"
Sep 11 18:00:09.555106 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:09.555063    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:00:12.646717 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:12.646683    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"
Sep 11 18:00:14.787436 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:14.787389    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:00:14.787874 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:14.787450    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:00:14.800282 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:14.800252    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:00:15.696144 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:15.696106    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:00:20.009118 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:20.009082    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"]
Sep 11 18:00:20.009583 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:20.009292    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49" podUID="9595f4ff-9001-424f-ba1e-b8fe8af1fa92" containerName="maas-api" containerID="cri-o://a59b2a4c95bda7bcbe7f205f0aea632901cd6dfb0f7b7acb6716478462d82e29" gracePeriod=30
Sep 11 18:00:20.263117 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:20.263048    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"
Sep 11 18:00:20.383704 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:20.383665    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-metrics-certs\") pod \"9595f4ff-9001-424f-ba1e-b8fe8af1fa92\" (UID: \"9595f4ff-9001-424f-ba1e-b8fe8af1fa92\") "
Sep 11 18:00:20.383903 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:20.383721    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-openshift-service-ca\") pod \"9595f4ff-9001-424f-ba1e-b8fe8af1fa92\" (UID: \"9595f4ff-9001-424f-ba1e-b8fe8af1fa92\") "
Sep 11 18:00:20.383903 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:20.383818    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmrsc\" (UniqueName: \"kubernetes.io/projected/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-kube-api-access-wmrsc\") pod \"9595f4ff-9001-424f-ba1e-b8fe8af1fa92\" (UID: \"9595f4ff-9001-424f-ba1e-b8fe8af1fa92\") "
Sep 11 18:00:20.383903 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:20.383855    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-maas-api-tls\") pod \"9595f4ff-9001-424f-ba1e-b8fe8af1fa92\" (UID: \"9595f4ff-9001-424f-ba1e-b8fe8af1fa92\") "
Sep 11 18:00:20.384212 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:20.384182    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9595f4ff-9001-424f-ba1e-b8fe8af1fa92" (UID: "9595f4ff-9001-424f-ba1e-b8fe8af1fa92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:00:20.385922 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:20.385894    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9595f4ff-9001-424f-ba1e-b8fe8af1fa92" (UID: "9595f4ff-9001-424f-ba1e-b8fe8af1fa92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:00:20.386021 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:20.385920    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9595f4ff-9001-424f-ba1e-b8fe8af1fa92" (UID: "9595f4ff-9001-424f-ba1e-b8fe8af1fa92"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:00:20.386021 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:20.385955    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-kube-api-access-wmrsc" (OuterVolumeSpecName: "kube-api-access-wmrsc") pod "9595f4ff-9001-424f-ba1e-b8fe8af1fa92" (UID: "9595f4ff-9001-424f-ba1e-b8fe8af1fa92"). InnerVolumeSpecName "kube-api-access-wmrsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:00:20.484864 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:20.484815    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-metrics-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:00:20.484864 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:20.484855    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-openshift-service-ca\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:00:20.484864 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:20.484864    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmrsc\" (UniqueName: \"kubernetes.io/projected/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-kube-api-access-wmrsc\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:00:20.484864 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:20.484873    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9595f4ff-9001-424f-ba1e-b8fe8af1fa92-maas-api-tls\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:00:20.712522 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:20.712487    2559 generic.go:356] "Generic (PLEG): container finished" podID="9595f4ff-9001-424f-ba1e-b8fe8af1fa92" containerID="a59b2a4c95bda7bcbe7f205f0aea632901cd6dfb0f7b7acb6716478462d82e29" exitCode=0
Sep 11 18:00:20.712703 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:20.712534    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"
Sep 11 18:00:20.712703 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:20.712567    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49" event={"ID":"9595f4ff-9001-424f-ba1e-b8fe8af1fa92","Type":"ContainerDied","Data":"a59b2a4c95bda7bcbe7f205f0aea632901cd6dfb0f7b7acb6716478462d82e29"}
Sep 11 18:00:20.712703 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:20.712619    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49" event={"ID":"9595f4ff-9001-424f-ba1e-b8fe8af1fa92","Type":"ContainerDied","Data":"36c82164bcbec0ab5132c667c7c0aa240ae50447464135d1369c13dc652ff491"}
Sep 11 18:00:20.712703 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:20.712642    2559 scope.go:117] "RemoveContainer" containerID="a59b2a4c95bda7bcbe7f205f0aea632901cd6dfb0f7b7acb6716478462d82e29"
Sep 11 18:00:20.722980 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:20.722963    2559 scope.go:117] "RemoveContainer" containerID="a59b2a4c95bda7bcbe7f205f0aea632901cd6dfb0f7b7acb6716478462d82e29"
Sep 11 18:00:20.723269 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:00:20.723251    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a59b2a4c95bda7bcbe7f205f0aea632901cd6dfb0f7b7acb6716478462d82e29\": container with ID starting with a59b2a4c95bda7bcbe7f205f0aea632901cd6dfb0f7b7acb6716478462d82e29 not found: ID does not exist" containerID="a59b2a4c95bda7bcbe7f205f0aea632901cd6dfb0f7b7acb6716478462d82e29"
Sep 11 18:00:20.723336 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:20.723280    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a59b2a4c95bda7bcbe7f205f0aea632901cd6dfb0f7b7acb6716478462d82e29"} err="failed to get container status \"a59b2a4c95bda7bcbe7f205f0aea632901cd6dfb0f7b7acb6716478462d82e29\": rpc error: code = NotFound desc = could not find container \"a59b2a4c95bda7bcbe7f205f0aea632901cd6dfb0f7b7acb6716478462d82e29\": container with ID starting with a59b2a4c95bda7bcbe7f205f0aea632901cd6dfb0f7b7acb6716478462d82e29 not found: ID does not exist"
Sep 11 18:00:20.743083 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:20.743051    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"]
Sep 11 18:00:20.746148 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:20.746119    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5883b7-7d5ff58b6f-g4c49"]
Sep 11 18:00:21.053072 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:21.052963    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9595f4ff-9001-424f-ba1e-b8fe8af1fa92" path="/var/lib/kubelet/pods/9595f4ff-9001-424f-ba1e-b8fe8af1fa92/volumes"
Sep 11 18:00:25.772011 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:25.771970    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"]
Sep 11 18:00:26.444380 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:26.444321    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:00:31.259102 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:31.259058    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"]
Sep 11 18:00:31.259529 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:31.259366    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm" podUID="2e6cded0-1fda-443d-a74a-b7c5a4f21e9b" containerName="maas-api" containerID="cri-o://b46de0002af367c6ff2a120f434fe3158984a7bb25be1d70828ab97582c84043" gracePeriod=30
Sep 11 18:00:31.508031 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:31.508001    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"
Sep 11 18:00:31.699907 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:31.699868    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-metrics-certs\") pod \"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b\" (UID: \"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b\") "
Sep 11 18:00:31.700086 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:31.699916    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-openshift-service-ca\") pod \"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b\" (UID: \"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b\") "
Sep 11 18:00:31.700086 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:31.699945    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hg9vh\" (UniqueName: \"kubernetes.io/projected/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-kube-api-access-hg9vh\") pod \"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b\" (UID: \"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b\") "
Sep 11 18:00:31.700086 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:31.700064    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-maas-api-tls\") pod \"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b\" (UID: \"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b\") "
Sep 11 18:00:31.700375 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:31.700323    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2e6cded0-1fda-443d-a74a-b7c5a4f21e9b" (UID: "2e6cded0-1fda-443d-a74a-b7c5a4f21e9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:00:31.700545 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:31.700527    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-openshift-service-ca\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:00:31.702202 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:31.702170    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2e6cded0-1fda-443d-a74a-b7c5a4f21e9b" (UID: "2e6cded0-1fda-443d-a74a-b7c5a4f21e9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:00:31.702308 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:31.702197    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2e6cded0-1fda-443d-a74a-b7c5a4f21e9b" (UID: "2e6cded0-1fda-443d-a74a-b7c5a4f21e9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:00:31.702308 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:31.702277    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-kube-api-access-hg9vh" (OuterVolumeSpecName: "kube-api-access-hg9vh") pod "2e6cded0-1fda-443d-a74a-b7c5a4f21e9b" (UID: "2e6cded0-1fda-443d-a74a-b7c5a4f21e9b"). InnerVolumeSpecName "kube-api-access-hg9vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:00:31.777728 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:31.777692    2559 generic.go:356] "Generic (PLEG): container finished" podID="2e6cded0-1fda-443d-a74a-b7c5a4f21e9b" containerID="b46de0002af367c6ff2a120f434fe3158984a7bb25be1d70828ab97582c84043" exitCode=0
Sep 11 18:00:31.777909 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:31.777735    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"
Sep 11 18:00:31.777909 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:31.777776    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm" event={"ID":"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b","Type":"ContainerDied","Data":"b46de0002af367c6ff2a120f434fe3158984a7bb25be1d70828ab97582c84043"}
Sep 11 18:00:31.777909 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:31.777825    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm" event={"ID":"2e6cded0-1fda-443d-a74a-b7c5a4f21e9b","Type":"ContainerDied","Data":"56ce9149959d4695a4d821054bbcdbdbefbd312a7f5d8eab1adfd1a9de6b9835"}
Sep 11 18:00:31.777909 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:31.777851    2559 scope.go:117] "RemoveContainer" containerID="b46de0002af367c6ff2a120f434fe3158984a7bb25be1d70828ab97582c84043"
Sep 11 18:00:31.787411 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:31.787390    2559 scope.go:117] "RemoveContainer" containerID="b46de0002af367c6ff2a120f434fe3158984a7bb25be1d70828ab97582c84043"
Sep 11 18:00:31.787719 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:00:31.787698    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b46de0002af367c6ff2a120f434fe3158984a7bb25be1d70828ab97582c84043\": container with ID starting with b46de0002af367c6ff2a120f434fe3158984a7bb25be1d70828ab97582c84043 not found: ID does not exist" containerID="b46de0002af367c6ff2a120f434fe3158984a7bb25be1d70828ab97582c84043"
Sep 11 18:00:31.787770 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:31.787730    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b46de0002af367c6ff2a120f434fe3158984a7bb25be1d70828ab97582c84043"} err="failed to get container status \"b46de0002af367c6ff2a120f434fe3158984a7bb25be1d70828ab97582c84043\": rpc error: code = NotFound desc = could not find container \"b46de0002af367c6ff2a120f434fe3158984a7bb25be1d70828ab97582c84043\": container with ID starting with b46de0002af367c6ff2a120f434fe3158984a7bb25be1d70828ab97582c84043 not found: ID does not exist"
Sep 11 18:00:31.800810 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:31.800774    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"]
Sep 11 18:00:31.801242 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:31.801219    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-metrics-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:00:31.801294 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:31.801246    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg9vh\" (UniqueName: \"kubernetes.io/projected/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-kube-api-access-hg9vh\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:00:31.801294 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:31.801257    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b-maas-api-tls\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:00:31.805818 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:31.805788    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c83b88f4-55bf4d4cf6-5w5rm"]
Sep 11 18:00:33.052783 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:33.052740    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e6cded0-1fda-443d-a74a-b7c5a4f21e9b" path="/var/lib/kubelet/pods/2e6cded0-1fda-443d-a74a-b7c5a4f21e9b/volumes"
Sep 11 18:00:37.143235 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:37.143200    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:00:40.791850 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:40.791776    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7" podUID="3b8bc1d7-09f9-48c2-94be-0ae310ca2a89" containerName="main" containerID="cri-o://ad4686bb0053c604bdb232520ddeb49e436c29a1b1ff8f5ceae137faba24c45c" gracePeriod=45
Sep 11 18:00:41.047731 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.047659    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 18:00:41.076985 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.076946    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-dshm\") pod \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\" (UID: \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\") "
Sep 11 18:00:41.077173 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.077003    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-tls-certs\") pod \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\" (UID: \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\") "
Sep 11 18:00:41.077173 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.077033    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-home\") pod \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\" (UID: \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\") "
Sep 11 18:00:41.077173 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.077102    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtll9\" (UniqueName: \"kubernetes.io/projected/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-kube-api-access-vtll9\") pod \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\" (UID: \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\") "
Sep 11 18:00:41.077376 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.077178    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-tmp-dir\") pod \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\" (UID: \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\") "
Sep 11 18:00:41.077376 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.077203    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-model-cache\") pod \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\" (UID: \"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89\") "
Sep 11 18:00:41.077494 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.077380    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-home" (OuterVolumeSpecName: "home") pod "3b8bc1d7-09f9-48c2-94be-0ae310ca2a89" (UID: "3b8bc1d7-09f9-48c2-94be-0ae310ca2a89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:00:41.077565 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.077461    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3b8bc1d7-09f9-48c2-94be-0ae310ca2a89" (UID: "3b8bc1d7-09f9-48c2-94be-0ae310ca2a89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:00:41.077565 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.077533    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-model-cache" (OuterVolumeSpecName: "model-cache") pod "3b8bc1d7-09f9-48c2-94be-0ae310ca2a89" (UID: "3b8bc1d7-09f9-48c2-94be-0ae310ca2a89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:00:41.077679 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.077637    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-home\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:00:41.077679 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.077658    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-tmp-dir\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:00:41.077679 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.077670    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-model-cache\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:00:41.079420 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.079393    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-kube-api-access-vtll9" (OuterVolumeSpecName: "kube-api-access-vtll9") pod "3b8bc1d7-09f9-48c2-94be-0ae310ca2a89" (UID: "3b8bc1d7-09f9-48c2-94be-0ae310ca2a89"). InnerVolumeSpecName "kube-api-access-vtll9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:00:41.079420 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.079399    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-dshm" (OuterVolumeSpecName: "dshm") pod "3b8bc1d7-09f9-48c2-94be-0ae310ca2a89" (UID: "3b8bc1d7-09f9-48c2-94be-0ae310ca2a89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:00:41.079574 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.079501    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b8bc1d7-09f9-48c2-94be-0ae310ca2a89" (UID: "3b8bc1d7-09f9-48c2-94be-0ae310ca2a89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:00:41.178300 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.178253    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-tls-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:00:41.178300 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.178293    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtll9\" (UniqueName: \"kubernetes.io/projected/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-kube-api-access-vtll9\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:00:41.178300 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.178302    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89-dshm\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:00:41.834738 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.834701    2559 generic.go:356] "Generic (PLEG): container finished" podID="3b8bc1d7-09f9-48c2-94be-0ae310ca2a89" containerID="ad4686bb0053c604bdb232520ddeb49e436c29a1b1ff8f5ceae137faba24c45c" exitCode=0
Sep 11 18:00:41.835160 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.834760    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"
Sep 11 18:00:41.835160 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.834785    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7" event={"ID":"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89","Type":"ContainerDied","Data":"ad4686bb0053c604bdb232520ddeb49e436c29a1b1ff8f5ceae137faba24c45c"}
Sep 11 18:00:41.835160 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.834829    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7" event={"ID":"3b8bc1d7-09f9-48c2-94be-0ae310ca2a89","Type":"ContainerDied","Data":"ee82da552b2c29e3e3da1d0c5553d65673d8f7e06c9bdc73aa71f18db09b66cc"}
Sep 11 18:00:41.835160 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.834847    2559 scope.go:117] "RemoveContainer" containerID="ad4686bb0053c604bdb232520ddeb49e436c29a1b1ff8f5ceae137faba24c45c"
Sep 11 18:00:41.845414 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.845395    2559 scope.go:117] "RemoveContainer" containerID="ad4686bb0053c604bdb232520ddeb49e436c29a1b1ff8f5ceae137faba24c45c"
Sep 11 18:00:41.845708 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:00:41.845685    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad4686bb0053c604bdb232520ddeb49e436c29a1b1ff8f5ceae137faba24c45c\": container with ID starting with ad4686bb0053c604bdb232520ddeb49e436c29a1b1ff8f5ceae137faba24c45c not found: ID does not exist" containerID="ad4686bb0053c604bdb232520ddeb49e436c29a1b1ff8f5ceae137faba24c45c"
Sep 11 18:00:41.845770 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.845715    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad4686bb0053c604bdb232520ddeb49e436c29a1b1ff8f5ceae137faba24c45c"} err="failed to get container status \"ad4686bb0053c604bdb232520ddeb49e436c29a1b1ff8f5ceae137faba24c45c\": rpc error: code = NotFound desc = could not find container \"ad4686bb0053c604bdb232520ddeb49e436c29a1b1ff8f5ceae137faba24c45c\": container with ID starting with ad4686bb0053c604bdb232520ddeb49e436c29a1b1ff8f5ceae137faba24c45c not found: ID does not exist"
Sep 11 18:00:41.859643 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.859615    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"]
Sep 11 18:00:41.865950 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:41.865923    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-5883b7/autores-model-5883b7-kserve-765d54dbd7-5szf7"]
Sep 11 18:00:43.053436 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:43.053399    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b8bc1d7-09f9-48c2-94be-0ae310ca2a89" path="/var/lib/kubelet/pods/3b8bc1d7-09f9-48c2-94be-0ae310ca2a89/volumes"
Sep 11 18:00:57.144742 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.144703    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"]
Sep 11 18:00:57.145130 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.145062    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b8bc1d7-09f9-48c2-94be-0ae310ca2a89" containerName="main"
Sep 11 18:00:57.145130 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.145074    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b8bc1d7-09f9-48c2-94be-0ae310ca2a89" containerName="main"
Sep 11 18:00:57.145130 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.145082    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9595f4ff-9001-424f-ba1e-b8fe8af1fa92" containerName="maas-api"
Sep 11 18:00:57.145130 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.145087    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9595f4ff-9001-424f-ba1e-b8fe8af1fa92" containerName="maas-api"
Sep 11 18:00:57.145130 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.145095    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22904b5e-730e-4d2f-b488-1f48cb9e2052" containerName="cleanup"
Sep 11 18:00:57.145130 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.145101    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="22904b5e-730e-4d2f-b488-1f48cb9e2052" containerName="cleanup"
Sep 11 18:00:57.145130 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.145107    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e6cded0-1fda-443d-a74a-b7c5a4f21e9b" containerName="maas-api"
Sep 11 18:00:57.145130 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.145112    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e6cded0-1fda-443d-a74a-b7c5a4f21e9b" containerName="maas-api"
Sep 11 18:00:57.145408 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.145189    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e6cded0-1fda-443d-a74a-b7c5a4f21e9b" containerName="maas-api"
Sep 11 18:00:57.145408 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.145195    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b8bc1d7-09f9-48c2-94be-0ae310ca2a89" containerName="main"
Sep 11 18:00:57.145408 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.145202    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="22904b5e-730e-4d2f-b488-1f48cb9e2052" containerName="cleanup"
Sep 11 18:00:57.145408 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.145209    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9595f4ff-9001-424f-ba1e-b8fe8af1fa92" containerName="maas-api"
Sep 11 18:00:57.149735 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.149717    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:00:57.156404 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.156384    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-87ec3d46-kserve-self-signed-certs\""
Sep 11 18:00:57.175728 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.175698    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"]
Sep 11 18:00:57.227304 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.227267    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac5018f9-02b2-40d0-890a-6e79f1b1480e-tmp-dir\") pod \"e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz\" (UID: \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\") " pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:00:57.227516 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.227319    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac5018f9-02b2-40d0-890a-6e79f1b1480e-dshm\") pod \"e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz\" (UID: \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\") " pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:00:57.227516 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.227427    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac5018f9-02b2-40d0-890a-6e79f1b1480e-home\") pod \"e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz\" (UID: \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\") " pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:00:57.227516 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.227485    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdr4j\" (UniqueName: \"kubernetes.io/projected/ac5018f9-02b2-40d0-890a-6e79f1b1480e-kube-api-access-zdr4j\") pod \"e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz\" (UID: \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\") " pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:00:57.227693 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.227581    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac5018f9-02b2-40d0-890a-6e79f1b1480e-model-cache\") pod \"e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz\" (UID: \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\") " pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:00:57.227693 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.227622    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac5018f9-02b2-40d0-890a-6e79f1b1480e-tls-certs\") pod \"e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz\" (UID: \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\") " pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:00:57.328279 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.328241    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac5018f9-02b2-40d0-890a-6e79f1b1480e-model-cache\") pod \"e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz\" (UID: \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\") " pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:00:57.328279 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.328286    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac5018f9-02b2-40d0-890a-6e79f1b1480e-tls-certs\") pod \"e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz\" (UID: \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\") " pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:00:57.328559 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.328306    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac5018f9-02b2-40d0-890a-6e79f1b1480e-tmp-dir\") pod \"e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz\" (UID: \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\") " pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:00:57.328559 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.328340    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac5018f9-02b2-40d0-890a-6e79f1b1480e-dshm\") pod \"e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz\" (UID: \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\") " pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:00:57.328559 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.328396    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac5018f9-02b2-40d0-890a-6e79f1b1480e-home\") pod \"e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz\" (UID: \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\") " pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:00:57.328559 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.328431    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdr4j\" (UniqueName: \"kubernetes.io/projected/ac5018f9-02b2-40d0-890a-6e79f1b1480e-kube-api-access-zdr4j\") pod \"e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz\" (UID: \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\") " pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:00:57.328777 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.328752    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac5018f9-02b2-40d0-890a-6e79f1b1480e-tmp-dir\") pod \"e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz\" (UID: \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\") " pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:00:57.328826 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.328785    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac5018f9-02b2-40d0-890a-6e79f1b1480e-model-cache\") pod \"e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz\" (UID: \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\") " pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:00:57.328860 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.328822    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac5018f9-02b2-40d0-890a-6e79f1b1480e-home\") pod \"e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz\" (UID: \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\") " pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:00:57.330747 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.330726    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac5018f9-02b2-40d0-890a-6e79f1b1480e-dshm\") pod \"e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz\" (UID: \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\") " pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:00:57.330857 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.330840    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac5018f9-02b2-40d0-890a-6e79f1b1480e-tls-certs\") pod \"e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz\" (UID: \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\") " pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:00:57.340667 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.340645    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdr4j\" (UniqueName: \"kubernetes.io/projected/ac5018f9-02b2-40d0-890a-6e79f1b1480e-kube-api-access-zdr4j\") pod \"e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz\" (UID: \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\") " pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:00:57.458113 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.458005    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:00:57.739239 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.739214    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"]
Sep 11 18:00:57.740779 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:00:57.740749    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac5018f9_02b2_40d0_890a_6e79f1b1480e.slice/crio-f1b229431a0eca05f8d6dcfcd159e0a41c772c6aaddf47c079e3cda63d69fc7c WatchSource:0}: Error finding container f1b229431a0eca05f8d6dcfcd159e0a41c772c6aaddf47c079e3cda63d69fc7c: Status 404 returned error can't find the container with id f1b229431a0eca05f8d6dcfcd159e0a41c772c6aaddf47c079e3cda63d69fc7c
Sep 11 18:00:57.926749 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.926707    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz" event={"ID":"ac5018f9-02b2-40d0-890a-6e79f1b1480e","Type":"ContainerStarted","Data":"396b782d035ed2088c4bc598ae37d67149717588be16da8b602bb461c47a4476"}
Sep 11 18:00:57.926749 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.926755    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz" event={"ID":"ac5018f9-02b2-40d0-890a-6e79f1b1480e","Type":"ContainerStarted","Data":"f1b229431a0eca05f8d6dcfcd159e0a41c772c6aaddf47c079e3cda63d69fc7c"}
Sep 11 18:00:57.962576 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:00:57.962514    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz" podStartSLOduration=0.962495204 podStartE2EDuration="962.495204ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:00:57.960105865 +0000 UTC m=+963.405206435" watchObservedRunningTime="2026-09-11 18:00:57.962495204 +0000 UTC m=+963.407595774"
Sep 11 18:01:01.753489 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:01.753449    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:01:07.458963 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:07.458916    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:01:07.458963 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:07.458970    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:01:07.472345 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:07.472304    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:01:07.992289 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:07.992256    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:01:17.459716 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:17.459630    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:01:26.490849 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:26.490809    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"]
Sep 11 18:01:28.193935 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:28.193898    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"]
Sep 11 18:01:32.246324 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:32.246286    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:01:34.205904 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.205867    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"]
Sep 11 18:01:34.209658 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.209639    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:01:34.213206 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.213178    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fb8c5d\"/\"default-dockercfg-qhmhc\""
Sep 11 18:01:34.213385 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.213211    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fb8c5d\"/\"openshift-service-ca.crt\""
Sep 11 18:01:34.213385 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.213234    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fb8c5d\"/\"test-model-fb8c5d-kserve-self-signed-certs\""
Sep 11 18:01:34.213514 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.213470    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fb8c5d\"/\"kube-root-ca.crt\""
Sep 11 18:01:34.225853 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.225827    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"]
Sep 11 18:01:34.293855 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.293821    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34d571d9-7de8-45f1-8779-e60153247d9d-tls-certs\") pod \"test-model-fb8c5d-kserve-844f86dd9c-kjmtk\" (UID: \"34d571d9-7de8-45f1-8779-e60153247d9d\") " pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:01:34.294028 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.293862    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34d571d9-7de8-45f1-8779-e60153247d9d-home\") pod \"test-model-fb8c5d-kserve-844f86dd9c-kjmtk\" (UID: \"34d571d9-7de8-45f1-8779-e60153247d9d\") " pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:01:34.294028 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.293926    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp5xz\" (UniqueName: \"kubernetes.io/projected/34d571d9-7de8-45f1-8779-e60153247d9d-kube-api-access-sp5xz\") pod \"test-model-fb8c5d-kserve-844f86dd9c-kjmtk\" (UID: \"34d571d9-7de8-45f1-8779-e60153247d9d\") " pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:01:34.294028 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.293952    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34d571d9-7de8-45f1-8779-e60153247d9d-tmp-dir\") pod \"test-model-fb8c5d-kserve-844f86dd9c-kjmtk\" (UID: \"34d571d9-7de8-45f1-8779-e60153247d9d\") " pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:01:34.294028 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.293999    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34d571d9-7de8-45f1-8779-e60153247d9d-model-cache\") pod \"test-model-fb8c5d-kserve-844f86dd9c-kjmtk\" (UID: \"34d571d9-7de8-45f1-8779-e60153247d9d\") " pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:01:34.294028 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.294019    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34d571d9-7de8-45f1-8779-e60153247d9d-dshm\") pod \"test-model-fb8c5d-kserve-844f86dd9c-kjmtk\" (UID: \"34d571d9-7de8-45f1-8779-e60153247d9d\") " pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:01:34.394720 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.394681    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34d571d9-7de8-45f1-8779-e60153247d9d-model-cache\") pod \"test-model-fb8c5d-kserve-844f86dd9c-kjmtk\" (UID: \"34d571d9-7de8-45f1-8779-e60153247d9d\") " pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:01:34.394720 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.394722    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34d571d9-7de8-45f1-8779-e60153247d9d-dshm\") pod \"test-model-fb8c5d-kserve-844f86dd9c-kjmtk\" (UID: \"34d571d9-7de8-45f1-8779-e60153247d9d\") " pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:01:34.395004 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.394813    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34d571d9-7de8-45f1-8779-e60153247d9d-tls-certs\") pod \"test-model-fb8c5d-kserve-844f86dd9c-kjmtk\" (UID: \"34d571d9-7de8-45f1-8779-e60153247d9d\") " pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:01:34.395004 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.394846    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34d571d9-7de8-45f1-8779-e60153247d9d-home\") pod \"test-model-fb8c5d-kserve-844f86dd9c-kjmtk\" (UID: \"34d571d9-7de8-45f1-8779-e60153247d9d\") " pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:01:34.395004 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.394895    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp5xz\" (UniqueName: \"kubernetes.io/projected/34d571d9-7de8-45f1-8779-e60153247d9d-kube-api-access-sp5xz\") pod \"test-model-fb8c5d-kserve-844f86dd9c-kjmtk\" (UID: \"34d571d9-7de8-45f1-8779-e60153247d9d\") " pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:01:34.395004 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.394928    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34d571d9-7de8-45f1-8779-e60153247d9d-tmp-dir\") pod \"test-model-fb8c5d-kserve-844f86dd9c-kjmtk\" (UID: \"34d571d9-7de8-45f1-8779-e60153247d9d\") " pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:01:34.395220 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.395165    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34d571d9-7de8-45f1-8779-e60153247d9d-model-cache\") pod \"test-model-fb8c5d-kserve-844f86dd9c-kjmtk\" (UID: \"34d571d9-7de8-45f1-8779-e60153247d9d\") " pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:01:34.395280 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.395235    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34d571d9-7de8-45f1-8779-e60153247d9d-home\") pod \"test-model-fb8c5d-kserve-844f86dd9c-kjmtk\" (UID: \"34d571d9-7de8-45f1-8779-e60153247d9d\") " pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:01:34.395332 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.395284    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34d571d9-7de8-45f1-8779-e60153247d9d-tmp-dir\") pod \"test-model-fb8c5d-kserve-844f86dd9c-kjmtk\" (UID: \"34d571d9-7de8-45f1-8779-e60153247d9d\") " pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:01:34.397016 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.396998    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34d571d9-7de8-45f1-8779-e60153247d9d-dshm\") pod \"test-model-fb8c5d-kserve-844f86dd9c-kjmtk\" (UID: \"34d571d9-7de8-45f1-8779-e60153247d9d\") " pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:01:34.397282 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.397266    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34d571d9-7de8-45f1-8779-e60153247d9d-tls-certs\") pod \"test-model-fb8c5d-kserve-844f86dd9c-kjmtk\" (UID: \"34d571d9-7de8-45f1-8779-e60153247d9d\") " pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:01:34.404798 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.404774    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp5xz\" (UniqueName: \"kubernetes.io/projected/34d571d9-7de8-45f1-8779-e60153247d9d-kube-api-access-sp5xz\") pod \"test-model-fb8c5d-kserve-844f86dd9c-kjmtk\" (UID: \"34d571d9-7de8-45f1-8779-e60153247d9d\") " pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:01:34.519665 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.519567    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:01:34.798537 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:34.798503    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"]
Sep 11 18:01:34.799914 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:01:34.799888    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34d571d9_7de8_45f1_8779_e60153247d9d.slice/crio-3dac3ae6f1934d0831dbb448f4fd63b1caeaa22b2cf457ec771ed070b092c53f WatchSource:0}: Error finding container 3dac3ae6f1934d0831dbb448f4fd63b1caeaa22b2cf457ec771ed070b092c53f: Status 404 returned error can't find the container with id 3dac3ae6f1934d0831dbb448f4fd63b1caeaa22b2cf457ec771ed070b092c53f
Sep 11 18:01:35.132116 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:35.132016    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk" event={"ID":"34d571d9-7de8-45f1-8779-e60153247d9d","Type":"ContainerStarted","Data":"98b8e724c676bfdd4aed90fda3dbf32b5dccefc74d8454841c730f3a0b165e0a"}
Sep 11 18:01:35.132116 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:35.132061    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk" event={"ID":"34d571d9-7de8-45f1-8779-e60153247d9d","Type":"ContainerStarted","Data":"3dac3ae6f1934d0831dbb448f4fd63b1caeaa22b2cf457ec771ed070b092c53f"}
Sep 11 18:01:35.160075 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:35.160024    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk" podStartSLOduration=1.160008204 podStartE2EDuration="1.160008204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:01:35.158109448 +0000 UTC m=+1000.603210031" watchObservedRunningTime="2026-09-11 18:01:35.160008204 +0000 UTC m=+1000.605108775"
Sep 11 18:01:41.510150 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:41.510101    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz" podUID="ac5018f9-02b2-40d0-890a-6e79f1b1480e" containerName="main" containerID="cri-o://396b782d035ed2088c4bc598ae37d67149717588be16da8b602bb461c47a4476" gracePeriod=45
Sep 11 18:01:41.758961 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:41.758936    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:01:41.867502 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:41.867414    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac5018f9-02b2-40d0-890a-6e79f1b1480e-tls-certs\") pod \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\" (UID: \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\") "
Sep 11 18:01:41.867652 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:41.867509    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac5018f9-02b2-40d0-890a-6e79f1b1480e-home\") pod \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\" (UID: \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\") "
Sep 11 18:01:41.867652 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:41.867610    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac5018f9-02b2-40d0-890a-6e79f1b1480e-dshm\") pod \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\" (UID: \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\") "
Sep 11 18:01:41.867772 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:41.867698    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac5018f9-02b2-40d0-890a-6e79f1b1480e-tmp-dir\") pod \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\" (UID: \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\") "
Sep 11 18:01:41.867772 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:41.867708    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac5018f9-02b2-40d0-890a-6e79f1b1480e-home" (OuterVolumeSpecName: "home") pod "ac5018f9-02b2-40d0-890a-6e79f1b1480e" (UID: "ac5018f9-02b2-40d0-890a-6e79f1b1480e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:01:41.867772 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:41.867720    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdr4j\" (UniqueName: \"kubernetes.io/projected/ac5018f9-02b2-40d0-890a-6e79f1b1480e-kube-api-access-zdr4j\") pod \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\" (UID: \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\") "
Sep 11 18:01:41.867772 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:41.867735    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac5018f9-02b2-40d0-890a-6e79f1b1480e-model-cache\") pod \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\" (UID: \"ac5018f9-02b2-40d0-890a-6e79f1b1480e\") "
Sep 11 18:01:41.867975 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:41.867886    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac5018f9-02b2-40d0-890a-6e79f1b1480e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ac5018f9-02b2-40d0-890a-6e79f1b1480e" (UID: "ac5018f9-02b2-40d0-890a-6e79f1b1480e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:01:41.868046 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:41.868016    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac5018f9-02b2-40d0-890a-6e79f1b1480e-model-cache" (OuterVolumeSpecName: "model-cache") pod "ac5018f9-02b2-40d0-890a-6e79f1b1480e" (UID: "ac5018f9-02b2-40d0-890a-6e79f1b1480e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:01:41.868134 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:41.868118    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac5018f9-02b2-40d0-890a-6e79f1b1480e-home\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:01:41.868196 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:41.868134    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac5018f9-02b2-40d0-890a-6e79f1b1480e-tmp-dir\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:01:41.868196 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:41.868144    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac5018f9-02b2-40d0-890a-6e79f1b1480e-model-cache\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:01:41.869772 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:41.869745    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac5018f9-02b2-40d0-890a-6e79f1b1480e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ac5018f9-02b2-40d0-890a-6e79f1b1480e" (UID: "ac5018f9-02b2-40d0-890a-6e79f1b1480e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:01:41.869772 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:41.869744    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac5018f9-02b2-40d0-890a-6e79f1b1480e-dshm" (OuterVolumeSpecName: "dshm") pod "ac5018f9-02b2-40d0-890a-6e79f1b1480e" (UID: "ac5018f9-02b2-40d0-890a-6e79f1b1480e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:01:41.869926 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:41.869786    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5018f9-02b2-40d0-890a-6e79f1b1480e-kube-api-access-zdr4j" (OuterVolumeSpecName: "kube-api-access-zdr4j") pod "ac5018f9-02b2-40d0-890a-6e79f1b1480e" (UID: "ac5018f9-02b2-40d0-890a-6e79f1b1480e"). InnerVolumeSpecName "kube-api-access-zdr4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:01:41.969565 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:41.969527    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac5018f9-02b2-40d0-890a-6e79f1b1480e-dshm\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:01:41.969565 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:41.969558    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdr4j\" (UniqueName: \"kubernetes.io/projected/ac5018f9-02b2-40d0-890a-6e79f1b1480e-kube-api-access-zdr4j\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:01:41.969565 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:41.969569    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac5018f9-02b2-40d0-890a-6e79f1b1480e-tls-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:01:42.172142 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:42.172108    2559 generic.go:356] "Generic (PLEG): container finished" podID="ac5018f9-02b2-40d0-890a-6e79f1b1480e" containerID="396b782d035ed2088c4bc598ae37d67149717588be16da8b602bb461c47a4476" exitCode=0
Sep 11 18:01:42.172374 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:42.172149    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"
Sep 11 18:01:42.172374 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:42.172196    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz" event={"ID":"ac5018f9-02b2-40d0-890a-6e79f1b1480e","Type":"ContainerDied","Data":"396b782d035ed2088c4bc598ae37d67149717588be16da8b602bb461c47a4476"}
Sep 11 18:01:42.172374 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:42.172242    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz" event={"ID":"ac5018f9-02b2-40d0-890a-6e79f1b1480e","Type":"ContainerDied","Data":"f1b229431a0eca05f8d6dcfcd159e0a41c772c6aaddf47c079e3cda63d69fc7c"}
Sep 11 18:01:42.172374 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:42.172263    2559 scope.go:117] "RemoveContainer" containerID="396b782d035ed2088c4bc598ae37d67149717588be16da8b602bb461c47a4476"
Sep 11 18:01:42.184559 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:42.184501    2559 scope.go:117] "RemoveContainer" containerID="396b782d035ed2088c4bc598ae37d67149717588be16da8b602bb461c47a4476"
Sep 11 18:01:42.185116 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:01:42.185096    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"396b782d035ed2088c4bc598ae37d67149717588be16da8b602bb461c47a4476\": container with ID starting with 396b782d035ed2088c4bc598ae37d67149717588be16da8b602bb461c47a4476 not found: ID does not exist" containerID="396b782d035ed2088c4bc598ae37d67149717588be16da8b602bb461c47a4476"
Sep 11 18:01:42.185207 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:42.185125    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396b782d035ed2088c4bc598ae37d67149717588be16da8b602bb461c47a4476"} err="failed to get container status \"396b782d035ed2088c4bc598ae37d67149717588be16da8b602bb461c47a4476\": rpc error: code = NotFound desc = could not find container \"396b782d035ed2088c4bc598ae37d67149717588be16da8b602bb461c47a4476\": container with ID starting with 396b782d035ed2088c4bc598ae37d67149717588be16da8b602bb461c47a4476 not found: ID does not exist"
Sep 11 18:01:42.203425 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:42.203385    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"]
Sep 11 18:01:42.210242 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:42.210218    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-87ec3d46-kserve-69d8644d55-d7lbz"]
Sep 11 18:01:43.053181 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:43.053148    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac5018f9-02b2-40d0-890a-6e79f1b1480e" path="/var/lib/kubelet/pods/ac5018f9-02b2-40d0-890a-6e79f1b1480e/volumes"
Sep 11 18:01:43.212787 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:43.212737    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2" podUID="9dae27d9-cc71-453d-8048-b9c350d3e95b" containerName="main" containerID="cri-o://ac7efa550dc8a9d5e6e716f4d0b37b24675a9c63e96733f52af444d8f9bff908" gracePeriod=45
Sep 11 18:01:43.471477 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:43.471449    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 18:01:43.583303 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:43.583222    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dae27d9-cc71-453d-8048-b9c350d3e95b-dshm\") pod \"9dae27d9-cc71-453d-8048-b9c350d3e95b\" (UID: \"9dae27d9-cc71-453d-8048-b9c350d3e95b\") "
Sep 11 18:01:43.583303 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:43.583273    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dae27d9-cc71-453d-8048-b9c350d3e95b-tmp-dir\") pod \"9dae27d9-cc71-453d-8048-b9c350d3e95b\" (UID: \"9dae27d9-cc71-453d-8048-b9c350d3e95b\") "
Sep 11 18:01:43.583504 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:43.583311    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dae27d9-cc71-453d-8048-b9c350d3e95b-model-cache\") pod \"9dae27d9-cc71-453d-8048-b9c350d3e95b\" (UID: \"9dae27d9-cc71-453d-8048-b9c350d3e95b\") "
Sep 11 18:01:43.583504 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:43.583438    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dae27d9-cc71-453d-8048-b9c350d3e95b-tls-certs\") pod \"9dae27d9-cc71-453d-8048-b9c350d3e95b\" (UID: \"9dae27d9-cc71-453d-8048-b9c350d3e95b\") "
Sep 11 18:01:43.583504 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:43.583482    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dae27d9-cc71-453d-8048-b9c350d3e95b-home\") pod \"9dae27d9-cc71-453d-8048-b9c350d3e95b\" (UID: \"9dae27d9-cc71-453d-8048-b9c350d3e95b\") "
Sep 11 18:01:43.583634 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:43.583514    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2p2w7\" (UniqueName: \"kubernetes.io/projected/9dae27d9-cc71-453d-8048-b9c350d3e95b-kube-api-access-2p2w7\") pod \"9dae27d9-cc71-453d-8048-b9c350d3e95b\" (UID: \"9dae27d9-cc71-453d-8048-b9c350d3e95b\") "
Sep 11 18:01:43.583634 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:43.583601    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dae27d9-cc71-453d-8048-b9c350d3e95b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9dae27d9-cc71-453d-8048-b9c350d3e95b" (UID: "9dae27d9-cc71-453d-8048-b9c350d3e95b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:01:43.583737 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:43.583650    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dae27d9-cc71-453d-8048-b9c350d3e95b-model-cache" (OuterVolumeSpecName: "model-cache") pod "9dae27d9-cc71-453d-8048-b9c350d3e95b" (UID: "9dae27d9-cc71-453d-8048-b9c350d3e95b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:01:43.583795 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:43.583742    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dae27d9-cc71-453d-8048-b9c350d3e95b-home" (OuterVolumeSpecName: "home") pod "9dae27d9-cc71-453d-8048-b9c350d3e95b" (UID: "9dae27d9-cc71-453d-8048-b9c350d3e95b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:01:43.583897 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:43.583880    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dae27d9-cc71-453d-8048-b9c350d3e95b-home\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:01:43.583954 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:43.583897    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dae27d9-cc71-453d-8048-b9c350d3e95b-tmp-dir\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:01:43.583954 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:43.583906    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dae27d9-cc71-453d-8048-b9c350d3e95b-model-cache\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:01:43.585487 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:43.585464    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dae27d9-cc71-453d-8048-b9c350d3e95b-dshm" (OuterVolumeSpecName: "dshm") pod "9dae27d9-cc71-453d-8048-b9c350d3e95b" (UID: "9dae27d9-cc71-453d-8048-b9c350d3e95b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:01:43.585579 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:43.585533    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dae27d9-cc71-453d-8048-b9c350d3e95b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9dae27d9-cc71-453d-8048-b9c350d3e95b" (UID: "9dae27d9-cc71-453d-8048-b9c350d3e95b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:01:43.585640 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:43.585593    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dae27d9-cc71-453d-8048-b9c350d3e95b-kube-api-access-2p2w7" (OuterVolumeSpecName: "kube-api-access-2p2w7") pod "9dae27d9-cc71-453d-8048-b9c350d3e95b" (UID: "9dae27d9-cc71-453d-8048-b9c350d3e95b"). InnerVolumeSpecName "kube-api-access-2p2w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:01:43.685373 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:43.685292    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dae27d9-cc71-453d-8048-b9c350d3e95b-tls-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:01:43.685373 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:43.685341    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p2w7\" (UniqueName: \"kubernetes.io/projected/9dae27d9-cc71-453d-8048-b9c350d3e95b-kube-api-access-2p2w7\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:01:43.685373 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:43.685381    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dae27d9-cc71-453d-8048-b9c350d3e95b-dshm\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:01:44.185603 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:44.185565    2559 generic.go:356] "Generic (PLEG): container finished" podID="9dae27d9-cc71-453d-8048-b9c350d3e95b" containerID="ac7efa550dc8a9d5e6e716f4d0b37b24675a9c63e96733f52af444d8f9bff908" exitCode=0
Sep 11 18:01:44.185603 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:44.185602    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"
Sep 11 18:01:44.186119 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:44.185651    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2" event={"ID":"9dae27d9-cc71-453d-8048-b9c350d3e95b","Type":"ContainerDied","Data":"ac7efa550dc8a9d5e6e716f4d0b37b24675a9c63e96733f52af444d8f9bff908"}
Sep 11 18:01:44.186119 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:44.185692    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2" event={"ID":"9dae27d9-cc71-453d-8048-b9c350d3e95b","Type":"ContainerDied","Data":"4fe6f572613adc73dad8a2cd358dede5e53365c052a7f8c547d02eec5dd8d276"}
Sep 11 18:01:44.186119 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:44.185710    2559 scope.go:117] "RemoveContainer" containerID="ac7efa550dc8a9d5e6e716f4d0b37b24675a9c63e96733f52af444d8f9bff908"
Sep 11 18:01:44.195031 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:44.195014    2559 scope.go:117] "RemoveContainer" containerID="ac7efa550dc8a9d5e6e716f4d0b37b24675a9c63e96733f52af444d8f9bff908"
Sep 11 18:01:44.195294 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:01:44.195271    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac7efa550dc8a9d5e6e716f4d0b37b24675a9c63e96733f52af444d8f9bff908\": container with ID starting with ac7efa550dc8a9d5e6e716f4d0b37b24675a9c63e96733f52af444d8f9bff908 not found: ID does not exist" containerID="ac7efa550dc8a9d5e6e716f4d0b37b24675a9c63e96733f52af444d8f9bff908"
Sep 11 18:01:44.195393 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:44.195302    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac7efa550dc8a9d5e6e716f4d0b37b24675a9c63e96733f52af444d8f9bff908"} err="failed to get container status \"ac7efa550dc8a9d5e6e716f4d0b37b24675a9c63e96733f52af444d8f9bff908\": rpc error: code = NotFound desc = could not find container \"ac7efa550dc8a9d5e6e716f4d0b37b24675a9c63e96733f52af444d8f9bff908\": container with ID starting with ac7efa550dc8a9d5e6e716f4d0b37b24675a9c63e96733f52af444d8f9bff908 not found: ID does not exist"
Sep 11 18:01:44.212037 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:44.212009    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"]
Sep 11 18:01:44.216437 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:44.216413    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-87ec3d46-kserve-67bc6f46bf-bkjd2"]
Sep 11 18:01:44.519933 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:44.519844    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:01:44.519933 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:44.519890    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:01:44.532577 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:44.532554    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:01:45.052738 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:45.052704    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dae27d9-cc71-453d-8048-b9c350d3e95b" path="/var/lib/kubelet/pods/9dae27d9-cc71-453d-8048-b9c350d3e95b/volumes"
Sep 11 18:01:45.203400 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:45.203344    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:01:46.159965 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:01:46.159927    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:02:03.481411 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.481378    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"]
Sep 11 18:02:03.481869 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.481756    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dae27d9-cc71-453d-8048-b9c350d3e95b" containerName="main"
Sep 11 18:02:03.481869 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.481768    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dae27d9-cc71-453d-8048-b9c350d3e95b" containerName="main"
Sep 11 18:02:03.481869 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.481792    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac5018f9-02b2-40d0-890a-6e79f1b1480e" containerName="main"
Sep 11 18:02:03.481869 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.481799    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac5018f9-02b2-40d0-890a-6e79f1b1480e" containerName="main"
Sep 11 18:02:03.481869 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.481853    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dae27d9-cc71-453d-8048-b9c350d3e95b" containerName="main"
Sep 11 18:02:03.481869 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.481865    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac5018f9-02b2-40d0-890a-6e79f1b1480e" containerName="main"
Sep 11 18:02:03.485746 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.485727    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"
Sep 11 18:02:03.489450 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.489429    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c69872\"/\"default-dockercfg-89ch5\""
Sep 11 18:02:03.489598 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.489460    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c69872\"/\"kube-root-ca.crt\""
Sep 11 18:02:03.489598 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.489431    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c69872\"/\"openshift-service-ca.crt\""
Sep 11 18:02:03.489598 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.489495    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c69872\"/\"explicit-model-c69872-kserve-self-signed-certs\""
Sep 11 18:02:03.500739 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.500716    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"]
Sep 11 18:02:03.570067 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.570035    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1417c31-f6c4-4269-ad3c-adc96506ab56-tls-certs\") pod \"explicit-model-c69872-kserve-659f449d4c-ltgt6\" (UID: \"f1417c31-f6c4-4269-ad3c-adc96506ab56\") " pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"
Sep 11 18:02:03.570261 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.570084    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1417c31-f6c4-4269-ad3c-adc96506ab56-dshm\") pod \"explicit-model-c69872-kserve-659f449d4c-ltgt6\" (UID: \"f1417c31-f6c4-4269-ad3c-adc96506ab56\") " pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"
Sep 11 18:02:03.570261 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.570103    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1417c31-f6c4-4269-ad3c-adc96506ab56-model-cache\") pod \"explicit-model-c69872-kserve-659f449d4c-ltgt6\" (UID: \"f1417c31-f6c4-4269-ad3c-adc96506ab56\") " pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"
Sep 11 18:02:03.570261 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.570147    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1417c31-f6c4-4269-ad3c-adc96506ab56-tmp-dir\") pod \"explicit-model-c69872-kserve-659f449d4c-ltgt6\" (UID: \"f1417c31-f6c4-4269-ad3c-adc96506ab56\") " pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"
Sep 11 18:02:03.570261 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.570178    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbt52\" (UniqueName: \"kubernetes.io/projected/f1417c31-f6c4-4269-ad3c-adc96506ab56-kube-api-access-lbt52\") pod \"explicit-model-c69872-kserve-659f449d4c-ltgt6\" (UID: \"f1417c31-f6c4-4269-ad3c-adc96506ab56\") " pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"
Sep 11 18:02:03.570261 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.570202    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1417c31-f6c4-4269-ad3c-adc96506ab56-home\") pod \"explicit-model-c69872-kserve-659f449d4c-ltgt6\" (UID: \"f1417c31-f6c4-4269-ad3c-adc96506ab56\") " pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"
Sep 11 18:02:03.671497 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.671456    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1417c31-f6c4-4269-ad3c-adc96506ab56-tmp-dir\") pod \"explicit-model-c69872-kserve-659f449d4c-ltgt6\" (UID: \"f1417c31-f6c4-4269-ad3c-adc96506ab56\") " pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"
Sep 11 18:02:03.671497 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.671503    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbt52\" (UniqueName: \"kubernetes.io/projected/f1417c31-f6c4-4269-ad3c-adc96506ab56-kube-api-access-lbt52\") pod \"explicit-model-c69872-kserve-659f449d4c-ltgt6\" (UID: \"f1417c31-f6c4-4269-ad3c-adc96506ab56\") " pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"
Sep 11 18:02:03.671717 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.671620    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1417c31-f6c4-4269-ad3c-adc96506ab56-home\") pod \"explicit-model-c69872-kserve-659f449d4c-ltgt6\" (UID: \"f1417c31-f6c4-4269-ad3c-adc96506ab56\") " pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"
Sep 11 18:02:03.671791 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.671772    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1417c31-f6c4-4269-ad3c-adc96506ab56-tls-certs\") pod \"explicit-model-c69872-kserve-659f449d4c-ltgt6\" (UID: \"f1417c31-f6c4-4269-ad3c-adc96506ab56\") " pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"
Sep 11 18:02:03.671872 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.671843    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1417c31-f6c4-4269-ad3c-adc96506ab56-dshm\") pod \"explicit-model-c69872-kserve-659f449d4c-ltgt6\" (UID: \"f1417c31-f6c4-4269-ad3c-adc96506ab56\") " pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"
Sep 11 18:02:03.672002 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.671892    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1417c31-f6c4-4269-ad3c-adc96506ab56-model-cache\") pod \"explicit-model-c69872-kserve-659f449d4c-ltgt6\" (UID: \"f1417c31-f6c4-4269-ad3c-adc96506ab56\") " pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"
Sep 11 18:02:03.672002 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.671910    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1417c31-f6c4-4269-ad3c-adc96506ab56-tmp-dir\") pod \"explicit-model-c69872-kserve-659f449d4c-ltgt6\" (UID: \"f1417c31-f6c4-4269-ad3c-adc96506ab56\") " pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"
Sep 11 18:02:03.672113 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.672032    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1417c31-f6c4-4269-ad3c-adc96506ab56-home\") pod \"explicit-model-c69872-kserve-659f449d4c-ltgt6\" (UID: \"f1417c31-f6c4-4269-ad3c-adc96506ab56\") " pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"
Sep 11 18:02:03.672243 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.672220    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1417c31-f6c4-4269-ad3c-adc96506ab56-model-cache\") pod \"explicit-model-c69872-kserve-659f449d4c-ltgt6\" (UID: \"f1417c31-f6c4-4269-ad3c-adc96506ab56\") " pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"
Sep 11 18:02:03.674004 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.673978    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1417c31-f6c4-4269-ad3c-adc96506ab56-dshm\") pod \"explicit-model-c69872-kserve-659f449d4c-ltgt6\" (UID: \"f1417c31-f6c4-4269-ad3c-adc96506ab56\") " pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"
Sep 11 18:02:03.674453 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.674433    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1417c31-f6c4-4269-ad3c-adc96506ab56-tls-certs\") pod \"explicit-model-c69872-kserve-659f449d4c-ltgt6\" (UID: \"f1417c31-f6c4-4269-ad3c-adc96506ab56\") " pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"
Sep 11 18:02:03.686327 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.686304    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbt52\" (UniqueName: \"kubernetes.io/projected/f1417c31-f6c4-4269-ad3c-adc96506ab56-kube-api-access-lbt52\") pod \"explicit-model-c69872-kserve-659f449d4c-ltgt6\" (UID: \"f1417c31-f6c4-4269-ad3c-adc96506ab56\") " pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"
Sep 11 18:02:03.795604 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:03.795513    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"
Sep 11 18:02:04.070777 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:04.070748    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"]
Sep 11 18:02:04.072337 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:02:04.072298    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1417c31_f6c4_4269_ad3c_adc96506ab56.slice/crio-e6f355d5ad3393dce2a1679b4fea856feb715481751fc669c455c7a39afac635 WatchSource:0}: Error finding container e6f355d5ad3393dce2a1679b4fea856feb715481751fc669c455c7a39afac635: Status 404 returned error can't find the container with id e6f355d5ad3393dce2a1679b4fea856feb715481751fc669c455c7a39afac635
Sep 11 18:02:04.246324 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:04.246286    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:02:04.294385 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:04.294327    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6" event={"ID":"f1417c31-f6c4-4269-ad3c-adc96506ab56","Type":"ContainerStarted","Data":"9d43928ec137e3265c106ad322106e1d82ac394c75e26a26c1271bde95b0bfed"}
Sep 11 18:02:04.294537 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:04.294394    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6" event={"ID":"f1417c31-f6c4-4269-ad3c-adc96506ab56","Type":"ContainerStarted","Data":"e6f355d5ad3393dce2a1679b4fea856feb715481751fc669c455c7a39afac635"}
Sep 11 18:02:04.326463 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:04.326371    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6" podStartSLOduration=1.326344575 podStartE2EDuration="1.326344575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:02:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:02:04.323806132 +0000 UTC m=+1029.768906714" watchObservedRunningTime="2026-09-11 18:02:04.326344575 +0000 UTC m=+1029.771445186"
Sep 11 18:02:06.325565 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:06.325532    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"]
Sep 11 18:02:13.796053 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:13.796012    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"
Sep 11 18:02:19.371845 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:19.371809    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:02:21.345175 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:21.345137    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6" podUID="f1417c31-f6c4-4269-ad3c-adc96506ab56" containerName="main" containerID="cri-o://9d43928ec137e3265c106ad322106e1d82ac394c75e26a26c1271bde95b0bfed" gracePeriod=45
Sep 11 18:02:21.617797 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:21.617769    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"
Sep 11 18:02:21.744811 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:21.744782    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbt52\" (UniqueName: \"kubernetes.io/projected/f1417c31-f6c4-4269-ad3c-adc96506ab56-kube-api-access-lbt52\") pod \"f1417c31-f6c4-4269-ad3c-adc96506ab56\" (UID: \"f1417c31-f6c4-4269-ad3c-adc96506ab56\") "
Sep 11 18:02:21.744811 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:21.744817    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1417c31-f6c4-4269-ad3c-adc96506ab56-home\") pod \"f1417c31-f6c4-4269-ad3c-adc96506ab56\" (UID: \"f1417c31-f6c4-4269-ad3c-adc96506ab56\") "
Sep 11 18:02:21.745047 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:21.744845    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1417c31-f6c4-4269-ad3c-adc96506ab56-tls-certs\") pod \"f1417c31-f6c4-4269-ad3c-adc96506ab56\" (UID: \"f1417c31-f6c4-4269-ad3c-adc96506ab56\") "
Sep 11 18:02:21.745047 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:21.744870    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1417c31-f6c4-4269-ad3c-adc96506ab56-dshm\") pod \"f1417c31-f6c4-4269-ad3c-adc96506ab56\" (UID: \"f1417c31-f6c4-4269-ad3c-adc96506ab56\") "
Sep 11 18:02:21.745047 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:21.744922    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1417c31-f6c4-4269-ad3c-adc96506ab56-tmp-dir\") pod \"f1417c31-f6c4-4269-ad3c-adc96506ab56\" (UID: \"f1417c31-f6c4-4269-ad3c-adc96506ab56\") "
Sep 11 18:02:21.745047 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:21.745007    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1417c31-f6c4-4269-ad3c-adc96506ab56-model-cache\") pod \"f1417c31-f6c4-4269-ad3c-adc96506ab56\" (UID: \"f1417c31-f6c4-4269-ad3c-adc96506ab56\") "
Sep 11 18:02:21.745260 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:21.745108    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1417c31-f6c4-4269-ad3c-adc96506ab56-home" (OuterVolumeSpecName: "home") pod "f1417c31-f6c4-4269-ad3c-adc96506ab56" (UID: "f1417c31-f6c4-4269-ad3c-adc96506ab56"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:02:21.745416 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:21.745249    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1417c31-f6c4-4269-ad3c-adc96506ab56-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f1417c31-f6c4-4269-ad3c-adc96506ab56" (UID: "f1417c31-f6c4-4269-ad3c-adc96506ab56"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:02:21.745416 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:21.745304    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1417c31-f6c4-4269-ad3c-adc96506ab56-home\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:02:21.745416 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:21.745315    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1417c31-f6c4-4269-ad3c-adc96506ab56-model-cache" (OuterVolumeSpecName: "model-cache") pod "f1417c31-f6c4-4269-ad3c-adc96506ab56" (UID: "f1417c31-f6c4-4269-ad3c-adc96506ab56"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:02:21.747170 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:21.747141    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1417c31-f6c4-4269-ad3c-adc96506ab56-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1417c31-f6c4-4269-ad3c-adc96506ab56" (UID: "f1417c31-f6c4-4269-ad3c-adc96506ab56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:02:21.747275 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:21.747178    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1417c31-f6c4-4269-ad3c-adc96506ab56-dshm" (OuterVolumeSpecName: "dshm") pod "f1417c31-f6c4-4269-ad3c-adc96506ab56" (UID: "f1417c31-f6c4-4269-ad3c-adc96506ab56"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:02:21.747275 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:21.747195    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1417c31-f6c4-4269-ad3c-adc96506ab56-kube-api-access-lbt52" (OuterVolumeSpecName: "kube-api-access-lbt52") pod "f1417c31-f6c4-4269-ad3c-adc96506ab56" (UID: "f1417c31-f6c4-4269-ad3c-adc96506ab56"). InnerVolumeSpecName "kube-api-access-lbt52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:02:21.846265 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:21.846175    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1417c31-f6c4-4269-ad3c-adc96506ab56-model-cache\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:02:21.846265 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:21.846206    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbt52\" (UniqueName: \"kubernetes.io/projected/f1417c31-f6c4-4269-ad3c-adc96506ab56-kube-api-access-lbt52\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:02:21.846265 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:21.846217    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1417c31-f6c4-4269-ad3c-adc96506ab56-tls-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:02:21.846265 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:21.846226    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1417c31-f6c4-4269-ad3c-adc96506ab56-dshm\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:02:21.846265 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:21.846235    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1417c31-f6c4-4269-ad3c-adc96506ab56-tmp-dir\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:02:22.389571 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:22.389540    2559 generic.go:356] "Generic (PLEG): container finished" podID="f1417c31-f6c4-4269-ad3c-adc96506ab56" containerID="9d43928ec137e3265c106ad322106e1d82ac394c75e26a26c1271bde95b0bfed" exitCode=0
Sep 11 18:02:22.390021 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:22.389587    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"
Sep 11 18:02:22.390021 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:22.389633    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6" event={"ID":"f1417c31-f6c4-4269-ad3c-adc96506ab56","Type":"ContainerDied","Data":"9d43928ec137e3265c106ad322106e1d82ac394c75e26a26c1271bde95b0bfed"}
Sep 11 18:02:22.390021 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:22.389687    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6" event={"ID":"f1417c31-f6c4-4269-ad3c-adc96506ab56","Type":"ContainerDied","Data":"e6f355d5ad3393dce2a1679b4fea856feb715481751fc669c455c7a39afac635"}
Sep 11 18:02:22.390021 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:22.389706    2559 scope.go:117] "RemoveContainer" containerID="9d43928ec137e3265c106ad322106e1d82ac394c75e26a26c1271bde95b0bfed"
Sep 11 18:02:22.399767 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:22.399749    2559 scope.go:117] "RemoveContainer" containerID="9d43928ec137e3265c106ad322106e1d82ac394c75e26a26c1271bde95b0bfed"
Sep 11 18:02:22.400072 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:02:22.400052    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d43928ec137e3265c106ad322106e1d82ac394c75e26a26c1271bde95b0bfed\": container with ID starting with 9d43928ec137e3265c106ad322106e1d82ac394c75e26a26c1271bde95b0bfed not found: ID does not exist" containerID="9d43928ec137e3265c106ad322106e1d82ac394c75e26a26c1271bde95b0bfed"
Sep 11 18:02:22.400116 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:22.400082    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d43928ec137e3265c106ad322106e1d82ac394c75e26a26c1271bde95b0bfed"} err="failed to get container status \"9d43928ec137e3265c106ad322106e1d82ac394c75e26a26c1271bde95b0bfed\": rpc error: code = NotFound desc = could not find container \"9d43928ec137e3265c106ad322106e1d82ac394c75e26a26c1271bde95b0bfed\": container with ID starting with 9d43928ec137e3265c106ad322106e1d82ac394c75e26a26c1271bde95b0bfed not found: ID does not exist"
Sep 11 18:02:22.430012 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:22.429985    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"]
Sep 11 18:02:22.440866 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:22.440842    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-c69872/explicit-model-c69872-kserve-659f449d4c-ltgt6"]
Sep 11 18:02:23.059575 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:23.059540    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1417c31-f6c4-4269-ad3c-adc96506ab56" path="/var/lib/kubelet/pods/f1417c31-f6c4-4269-ad3c-adc96506ab56/volumes"
Sep 11 18:02:38.195738 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:38.195701    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:02:53.264109 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:53.264031    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:02:54.215309 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.215271    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"]
Sep 11 18:02:54.215670 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.215657    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1417c31-f6c4-4269-ad3c-adc96506ab56" containerName="main"
Sep 11 18:02:54.215716 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.215672    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1417c31-f6c4-4269-ad3c-adc96506ab56" containerName="main"
Sep 11 18:02:54.215750 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.215740    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1417c31-f6c4-4269-ad3c-adc96506ab56" containerName="main"
Sep 11 18:02:54.218883 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.218865    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"
Sep 11 18:02:54.225563 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.225111    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c0ad9c\"/\"kube-root-ca.crt\""
Sep 11 18:02:54.225563 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.225193    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c0ad9c\"/\"default-dockercfg-l2q2w\""
Sep 11 18:02:54.225563 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.225281    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c0ad9c\"/\"openshift-service-ca.crt\""
Sep 11 18:02:54.225563 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.225322    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c0ad9c\"/\"noten-model-c0ad9c-kserve-self-signed-certs\""
Sep 11 18:02:54.243972 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.243940    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"]
Sep 11 18:02:54.250794 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.250772    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-tls-certs\") pod \"noten-model-c0ad9c-kserve-798d97457f-q8g98\" (UID: \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\") " pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"
Sep 11 18:02:54.250897 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.250816    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-dshm\") pod \"noten-model-c0ad9c-kserve-798d97457f-q8g98\" (UID: \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\") " pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"
Sep 11 18:02:54.250897 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.250843    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-model-cache\") pod \"noten-model-c0ad9c-kserve-798d97457f-q8g98\" (UID: \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\") " pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"
Sep 11 18:02:54.250977 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.250928    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-home\") pod \"noten-model-c0ad9c-kserve-798d97457f-q8g98\" (UID: \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\") " pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"
Sep 11 18:02:54.251048 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.251026    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-tmp-dir\") pod \"noten-model-c0ad9c-kserve-798d97457f-q8g98\" (UID: \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\") " pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"
Sep 11 18:02:54.251116 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.251101    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tzxd\" (UniqueName: \"kubernetes.io/projected/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-kube-api-access-8tzxd\") pod \"noten-model-c0ad9c-kserve-798d97457f-q8g98\" (UID: \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\") " pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"
Sep 11 18:02:54.351716 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.351677    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-model-cache\") pod \"noten-model-c0ad9c-kserve-798d97457f-q8g98\" (UID: \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\") " pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"
Sep 11 18:02:54.352160 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.351729    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-home\") pod \"noten-model-c0ad9c-kserve-798d97457f-q8g98\" (UID: \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\") " pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"
Sep 11 18:02:54.352160 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.351771    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-tmp-dir\") pod \"noten-model-c0ad9c-kserve-798d97457f-q8g98\" (UID: \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\") " pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"
Sep 11 18:02:54.352160 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.351800    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tzxd\" (UniqueName: \"kubernetes.io/projected/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-kube-api-access-8tzxd\") pod \"noten-model-c0ad9c-kserve-798d97457f-q8g98\" (UID: \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\") " pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"
Sep 11 18:02:54.352160 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.351828    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-tls-certs\") pod \"noten-model-c0ad9c-kserve-798d97457f-q8g98\" (UID: \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\") " pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"
Sep 11 18:02:54.352160 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.351871    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-dshm\") pod \"noten-model-c0ad9c-kserve-798d97457f-q8g98\" (UID: \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\") " pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"
Sep 11 18:02:54.352160 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.352074    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-model-cache\") pod \"noten-model-c0ad9c-kserve-798d97457f-q8g98\" (UID: \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\") " pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"
Sep 11 18:02:54.352160 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.352145    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-tmp-dir\") pod \"noten-model-c0ad9c-kserve-798d97457f-q8g98\" (UID: \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\") " pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"
Sep 11 18:02:54.352486 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.352245    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-home\") pod \"noten-model-c0ad9c-kserve-798d97457f-q8g98\" (UID: \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\") " pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"
Sep 11 18:02:54.354180 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.354163    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-dshm\") pod \"noten-model-c0ad9c-kserve-798d97457f-q8g98\" (UID: \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\") " pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"
Sep 11 18:02:54.354305 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.354287    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-tls-certs\") pod \"noten-model-c0ad9c-kserve-798d97457f-q8g98\" (UID: \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\") " pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"
Sep 11 18:02:54.368307 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.368284    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tzxd\" (UniqueName: \"kubernetes.io/projected/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-kube-api-access-8tzxd\") pod \"noten-model-c0ad9c-kserve-798d97457f-q8g98\" (UID: \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\") " pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"
Sep 11 18:02:54.528308 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.528223    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"
Sep 11 18:02:54.850260 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:54.850230    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"]
Sep 11 18:02:54.851361 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:02:54.851314    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3f40e3b_f9e8_499e_aff1_b3e5d7027bbf.slice/crio-9931b44116d9ae63a7a33882800720fbae49e3c97af78641570099a4fbe02556 WatchSource:0}: Error finding container 9931b44116d9ae63a7a33882800720fbae49e3c97af78641570099a4fbe02556: Status 404 returned error can't find the container with id 9931b44116d9ae63a7a33882800720fbae49e3c97af78641570099a4fbe02556
Sep 11 18:02:55.573159 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:55.573077    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98" event={"ID":"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf","Type":"ContainerStarted","Data":"0415f7586523de526150e2d1eb4bb04734bfa384ed8718ec8fc6a223806837bb"}
Sep 11 18:02:55.573159 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:55.573118    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98" event={"ID":"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf","Type":"ContainerStarted","Data":"9931b44116d9ae63a7a33882800720fbae49e3c97af78641570099a4fbe02556"}
Sep 11 18:02:55.601180 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:55.601094    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98" podStartSLOduration=1.601077675 podStartE2EDuration="1.601077675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:02:55.599960988 +0000 UTC m=+1081.045061580" watchObservedRunningTime="2026-09-11 18:02:55.601077675 +0000 UTC m=+1081.046178316"
Sep 11 18:02:58.217258 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:02:58.217223    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"]
Sep 11 18:03:00.125795 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:00.125747    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"]
Sep 11 18:03:01.185548 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:01.185512    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"]
Sep 11 18:03:04.528996 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:04.528952    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"
Sep 11 18:03:09.817974 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:09.817940    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:03:13.236147 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.236090    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98" podUID="f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf" containerName="main" containerID="cri-o://0415f7586523de526150e2d1eb4bb04734bfa384ed8718ec8fc6a223806837bb" gracePeriod=45
Sep 11 18:03:13.487796 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.487728    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"
Sep 11 18:03:13.647594 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.647545    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tzxd\" (UniqueName: \"kubernetes.io/projected/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-kube-api-access-8tzxd\") pod \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\" (UID: \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\") "
Sep 11 18:03:13.647594 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.647598    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-tmp-dir\") pod \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\" (UID: \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\") "
Sep 11 18:03:13.647853 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.647648    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-home\") pod \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\" (UID: \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\") "
Sep 11 18:03:13.647853 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.647679    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-dshm\") pod \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\" (UID: \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\") "
Sep 11 18:03:13.647853 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.647698    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-tls-certs\") pod \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\" (UID: \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\") "
Sep 11 18:03:13.647853 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.647723    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-model-cache\") pod \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\" (UID: \"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf\") "
Sep 11 18:03:13.648072 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.647917    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-home" (OuterVolumeSpecName: "home") pod "f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf" (UID: "f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:03:13.648072 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.647927    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf" (UID: "f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:03:13.648072 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.648039    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf" (UID: "f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:03:13.648189 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.648156    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-home\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:03:13.648189 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.648175    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-model-cache\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:03:13.648273 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.648191    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-tmp-dir\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:03:13.649942 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.649916    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-dshm" (OuterVolumeSpecName: "dshm") pod "f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf" (UID: "f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:03:13.650066 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.649972    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf" (UID: "f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:03:13.650066 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.649996    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-kube-api-access-8tzxd" (OuterVolumeSpecName: "kube-api-access-8tzxd") pod "f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf" (UID: "f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf"). InnerVolumeSpecName "kube-api-access-8tzxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:03:13.671203 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.671175    2559 generic.go:356] "Generic (PLEG): container finished" podID="f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf" containerID="0415f7586523de526150e2d1eb4bb04734bfa384ed8718ec8fc6a223806837bb" exitCode=0
Sep 11 18:03:13.671368 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.671216    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"
Sep 11 18:03:13.671368 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.671264    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98" event={"ID":"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf","Type":"ContainerDied","Data":"0415f7586523de526150e2d1eb4bb04734bfa384ed8718ec8fc6a223806837bb"}
Sep 11 18:03:13.671368 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.671310    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98" event={"ID":"f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf","Type":"ContainerDied","Data":"9931b44116d9ae63a7a33882800720fbae49e3c97af78641570099a4fbe02556"}
Sep 11 18:03:13.671368 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.671335    2559 scope.go:117] "RemoveContainer" containerID="0415f7586523de526150e2d1eb4bb04734bfa384ed8718ec8fc6a223806837bb"
Sep 11 18:03:13.680958 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.680931    2559 scope.go:117] "RemoveContainer" containerID="0415f7586523de526150e2d1eb4bb04734bfa384ed8718ec8fc6a223806837bb"
Sep 11 18:03:13.681234 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:03:13.681214    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0415f7586523de526150e2d1eb4bb04734bfa384ed8718ec8fc6a223806837bb\": container with ID starting with 0415f7586523de526150e2d1eb4bb04734bfa384ed8718ec8fc6a223806837bb not found: ID does not exist" containerID="0415f7586523de526150e2d1eb4bb04734bfa384ed8718ec8fc6a223806837bb"
Sep 11 18:03:13.681301 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.681243    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0415f7586523de526150e2d1eb4bb04734bfa384ed8718ec8fc6a223806837bb"} err="failed to get container status \"0415f7586523de526150e2d1eb4bb04734bfa384ed8718ec8fc6a223806837bb\": rpc error: code = NotFound desc = could not find container \"0415f7586523de526150e2d1eb4bb04734bfa384ed8718ec8fc6a223806837bb\": container with ID starting with 0415f7586523de526150e2d1eb4bb04734bfa384ed8718ec8fc6a223806837bb not found: ID does not exist"
Sep 11 18:03:13.708775 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.708746    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"]
Sep 11 18:03:13.726980 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.726956    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-c0ad9c/noten-model-c0ad9c-kserve-798d97457f-q8g98"]
Sep 11 18:03:13.748729 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.748661    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tzxd\" (UniqueName: \"kubernetes.io/projected/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-kube-api-access-8tzxd\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:03:13.748729 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.748687    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-dshm\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:03:13.748729 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:13.748698    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf-tls-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:03:15.059482 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.059448    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf" path="/var/lib/kubelet/pods/f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf/volumes"
Sep 11 18:03:15.145134 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.145066    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k" podUID="28b282f7-eeab-422d-b460-0c927e390749" containerName="main" containerID="cri-o://e1fffabeaae41bbafbca9bea76272474b1424cdb48f61a1d286212a76115abcb" gracePeriod=45
Sep 11 18:03:15.398030 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.397999    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:03:15.465397 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.465365    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b282f7-eeab-422d-b460-0c927e390749-dshm\") pod \"28b282f7-eeab-422d-b460-0c927e390749\" (UID: \"28b282f7-eeab-422d-b460-0c927e390749\") "
Sep 11 18:03:15.465575 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.465415    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b282f7-eeab-422d-b460-0c927e390749-tls-certs\") pod \"28b282f7-eeab-422d-b460-0c927e390749\" (UID: \"28b282f7-eeab-422d-b460-0c927e390749\") "
Sep 11 18:03:15.465575 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.465446    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28b282f7-eeab-422d-b460-0c927e390749-home\") pod \"28b282f7-eeab-422d-b460-0c927e390749\" (UID: \"28b282f7-eeab-422d-b460-0c927e390749\") "
Sep 11 18:03:15.465575 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.465474    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6l75c\" (UniqueName: \"kubernetes.io/projected/28b282f7-eeab-422d-b460-0c927e390749-kube-api-access-6l75c\") pod \"28b282f7-eeab-422d-b460-0c927e390749\" (UID: \"28b282f7-eeab-422d-b460-0c927e390749\") "
Sep 11 18:03:15.465575 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.465539    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b282f7-eeab-422d-b460-0c927e390749-model-cache\") pod \"28b282f7-eeab-422d-b460-0c927e390749\" (UID: \"28b282f7-eeab-422d-b460-0c927e390749\") "
Sep 11 18:03:15.465788 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.465718    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b282f7-eeab-422d-b460-0c927e390749-tmp-dir\") pod \"28b282f7-eeab-422d-b460-0c927e390749\" (UID: \"28b282f7-eeab-422d-b460-0c927e390749\") "
Sep 11 18:03:15.465788 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.465707    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b282f7-eeab-422d-b460-0c927e390749-home" (OuterVolumeSpecName: "home") pod "28b282f7-eeab-422d-b460-0c927e390749" (UID: "28b282f7-eeab-422d-b460-0c927e390749"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:03:15.465891 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.465866    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b282f7-eeab-422d-b460-0c927e390749-model-cache" (OuterVolumeSpecName: "model-cache") pod "28b282f7-eeab-422d-b460-0c927e390749" (UID: "28b282f7-eeab-422d-b460-0c927e390749"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:03:15.466016 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.465993    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b282f7-eeab-422d-b460-0c927e390749-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "28b282f7-eeab-422d-b460-0c927e390749" (UID: "28b282f7-eeab-422d-b460-0c927e390749"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:03:15.466113 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.466090    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b282f7-eeab-422d-b460-0c927e390749-tmp-dir\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:03:15.466165 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.466114    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28b282f7-eeab-422d-b460-0c927e390749-home\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:03:15.466165 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.466128    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b282f7-eeab-422d-b460-0c927e390749-model-cache\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:03:15.467578 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.467559    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b282f7-eeab-422d-b460-0c927e390749-dshm" (OuterVolumeSpecName: "dshm") pod "28b282f7-eeab-422d-b460-0c927e390749" (UID: "28b282f7-eeab-422d-b460-0c927e390749"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:03:15.467970 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.467952    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b282f7-eeab-422d-b460-0c927e390749-kube-api-access-6l75c" (OuterVolumeSpecName: "kube-api-access-6l75c") pod "28b282f7-eeab-422d-b460-0c927e390749" (UID: "28b282f7-eeab-422d-b460-0c927e390749"). InnerVolumeSpecName "kube-api-access-6l75c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:03:15.468027 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.467952    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28b282f7-eeab-422d-b460-0c927e390749-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "28b282f7-eeab-422d-b460-0c927e390749" (UID: "28b282f7-eeab-422d-b460-0c927e390749"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:03:15.566692 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.566602    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b282f7-eeab-422d-b460-0c927e390749-dshm\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:03:15.566692 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.566636    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b282f7-eeab-422d-b460-0c927e390749-tls-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:03:15.566692 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.566648    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l75c\" (UniqueName: \"kubernetes.io/projected/28b282f7-eeab-422d-b460-0c927e390749-kube-api-access-6l75c\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:03:15.684704 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.684664    2559 generic.go:356] "Generic (PLEG): container finished" podID="28b282f7-eeab-422d-b460-0c927e390749" containerID="e1fffabeaae41bbafbca9bea76272474b1424cdb48f61a1d286212a76115abcb" exitCode=0
Sep 11 18:03:15.684870 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.684709    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"
Sep 11 18:03:15.684870 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.684709    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k" event={"ID":"28b282f7-eeab-422d-b460-0c927e390749","Type":"ContainerDied","Data":"e1fffabeaae41bbafbca9bea76272474b1424cdb48f61a1d286212a76115abcb"}
Sep 11 18:03:15.684870 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.684830    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k" event={"ID":"28b282f7-eeab-422d-b460-0c927e390749","Type":"ContainerDied","Data":"39fac2d93e3ae968b347b8974108dbe2ba43a0e81391e3cdf44f43634f0a3771"}
Sep 11 18:03:15.684870 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.684856    2559 scope.go:117] "RemoveContainer" containerID="e1fffabeaae41bbafbca9bea76272474b1424cdb48f61a1d286212a76115abcb"
Sep 11 18:03:15.694686 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.694667    2559 scope.go:117] "RemoveContainer" containerID="e1fffabeaae41bbafbca9bea76272474b1424cdb48f61a1d286212a76115abcb"
Sep 11 18:03:15.694939 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:03:15.694919    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1fffabeaae41bbafbca9bea76272474b1424cdb48f61a1d286212a76115abcb\": container with ID starting with e1fffabeaae41bbafbca9bea76272474b1424cdb48f61a1d286212a76115abcb not found: ID does not exist" containerID="e1fffabeaae41bbafbca9bea76272474b1424cdb48f61a1d286212a76115abcb"
Sep 11 18:03:15.694988 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.694947    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1fffabeaae41bbafbca9bea76272474b1424cdb48f61a1d286212a76115abcb"} err="failed to get container status \"e1fffabeaae41bbafbca9bea76272474b1424cdb48f61a1d286212a76115abcb\": rpc error: code = NotFound desc = could not find container \"e1fffabeaae41bbafbca9bea76272474b1424cdb48f61a1d286212a76115abcb\": container with ID starting with e1fffabeaae41bbafbca9bea76272474b1424cdb48f61a1d286212a76115abcb not found: ID does not exist"
Sep 11 18:03:15.720120 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.720092    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"]
Sep 11 18:03:15.741540 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:15.741505    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-67477b/test-model-67477b-kserve-55ccb45c87-qmk7k"]
Sep 11 18:03:16.204320 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.204264    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk" podUID="34d571d9-7de8-45f1-8779-e60153247d9d" containerName="main" containerID="cri-o://98b8e724c676bfdd4aed90fda3dbf32b5dccefc74d8454841c730f3a0b165e0a" gracePeriod=45
Sep 11 18:03:16.452062 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.452037    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:03:16.472871 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.472804    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34d571d9-7de8-45f1-8779-e60153247d9d-tmp-dir\") pod \"34d571d9-7de8-45f1-8779-e60153247d9d\" (UID: \"34d571d9-7de8-45f1-8779-e60153247d9d\") "
Sep 11 18:03:16.472977 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.472873    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34d571d9-7de8-45f1-8779-e60153247d9d-tls-certs\") pod \"34d571d9-7de8-45f1-8779-e60153247d9d\" (UID: \"34d571d9-7de8-45f1-8779-e60153247d9d\") "
Sep 11 18:03:16.472977 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.472904    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34d571d9-7de8-45f1-8779-e60153247d9d-home\") pod \"34d571d9-7de8-45f1-8779-e60153247d9d\" (UID: \"34d571d9-7de8-45f1-8779-e60153247d9d\") "
Sep 11 18:03:16.472977 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.472933    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34d571d9-7de8-45f1-8779-e60153247d9d-dshm\") pod \"34d571d9-7de8-45f1-8779-e60153247d9d\" (UID: \"34d571d9-7de8-45f1-8779-e60153247d9d\") "
Sep 11 18:03:16.472977 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.472952    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sp5xz\" (UniqueName: \"kubernetes.io/projected/34d571d9-7de8-45f1-8779-e60153247d9d-kube-api-access-sp5xz\") pod \"34d571d9-7de8-45f1-8779-e60153247d9d\" (UID: \"34d571d9-7de8-45f1-8779-e60153247d9d\") "
Sep 11 18:03:16.472977 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.472967    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34d571d9-7de8-45f1-8779-e60153247d9d-model-cache\") pod \"34d571d9-7de8-45f1-8779-e60153247d9d\" (UID: \"34d571d9-7de8-45f1-8779-e60153247d9d\") "
Sep 11 18:03:16.473240 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.473047    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d571d9-7de8-45f1-8779-e60153247d9d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "34d571d9-7de8-45f1-8779-e60153247d9d" (UID: "34d571d9-7de8-45f1-8779-e60153247d9d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:03:16.473240 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.473194    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d571d9-7de8-45f1-8779-e60153247d9d-model-cache" (OuterVolumeSpecName: "model-cache") pod "34d571d9-7de8-45f1-8779-e60153247d9d" (UID: "34d571d9-7de8-45f1-8779-e60153247d9d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:03:16.473240 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.473224    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d571d9-7de8-45f1-8779-e60153247d9d-home" (OuterVolumeSpecName: "home") pod "34d571d9-7de8-45f1-8779-e60153247d9d" (UID: "34d571d9-7de8-45f1-8779-e60153247d9d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:03:16.473375 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.473240    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34d571d9-7de8-45f1-8779-e60153247d9d-tmp-dir\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:03:16.475162 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.475135    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34d571d9-7de8-45f1-8779-e60153247d9d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34d571d9-7de8-45f1-8779-e60153247d9d" (UID: "34d571d9-7de8-45f1-8779-e60153247d9d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:03:16.475265 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.475182    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d571d9-7de8-45f1-8779-e60153247d9d-dshm" (OuterVolumeSpecName: "dshm") pod "34d571d9-7de8-45f1-8779-e60153247d9d" (UID: "34d571d9-7de8-45f1-8779-e60153247d9d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:03:16.475265 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.475226    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d571d9-7de8-45f1-8779-e60153247d9d-kube-api-access-sp5xz" (OuterVolumeSpecName: "kube-api-access-sp5xz") pod "34d571d9-7de8-45f1-8779-e60153247d9d" (UID: "34d571d9-7de8-45f1-8779-e60153247d9d"). InnerVolumeSpecName "kube-api-access-sp5xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:03:16.574559 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.574512    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34d571d9-7de8-45f1-8779-e60153247d9d-tls-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:03:16.574559 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.574550    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34d571d9-7de8-45f1-8779-e60153247d9d-home\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:03:16.574559 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.574559    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34d571d9-7de8-45f1-8779-e60153247d9d-dshm\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:03:16.574559 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.574568    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp5xz\" (UniqueName: \"kubernetes.io/projected/34d571d9-7de8-45f1-8779-e60153247d9d-kube-api-access-sp5xz\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:03:16.574824 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.574577    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34d571d9-7de8-45f1-8779-e60153247d9d-model-cache\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:03:16.691213 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.691172    2559 generic.go:356] "Generic (PLEG): container finished" podID="34d571d9-7de8-45f1-8779-e60153247d9d" containerID="98b8e724c676bfdd4aed90fda3dbf32b5dccefc74d8454841c730f3a0b165e0a" exitCode=0
Sep 11 18:03:16.691213 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.691212    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"
Sep 11 18:03:16.691481 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.691253    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk" event={"ID":"34d571d9-7de8-45f1-8779-e60153247d9d","Type":"ContainerDied","Data":"98b8e724c676bfdd4aed90fda3dbf32b5dccefc74d8454841c730f3a0b165e0a"}
Sep 11 18:03:16.691481 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.691289    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk" event={"ID":"34d571d9-7de8-45f1-8779-e60153247d9d","Type":"ContainerDied","Data":"3dac3ae6f1934d0831dbb448f4fd63b1caeaa22b2cf457ec771ed070b092c53f"}
Sep 11 18:03:16.691481 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.691308    2559 scope.go:117] "RemoveContainer" containerID="98b8e724c676bfdd4aed90fda3dbf32b5dccefc74d8454841c730f3a0b165e0a"
Sep 11 18:03:16.700837 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.700821    2559 scope.go:117] "RemoveContainer" containerID="98b8e724c676bfdd4aed90fda3dbf32b5dccefc74d8454841c730f3a0b165e0a"
Sep 11 18:03:16.701106 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:03:16.701087    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98b8e724c676bfdd4aed90fda3dbf32b5dccefc74d8454841c730f3a0b165e0a\": container with ID starting with 98b8e724c676bfdd4aed90fda3dbf32b5dccefc74d8454841c730f3a0b165e0a not found: ID does not exist" containerID="98b8e724c676bfdd4aed90fda3dbf32b5dccefc74d8454841c730f3a0b165e0a"
Sep 11 18:03:16.701158 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:16.701114    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b8e724c676bfdd4aed90fda3dbf32b5dccefc74d8454841c730f3a0b165e0a"} err="failed to get container status \"98b8e724c676bfdd4aed90fda3dbf32b5dccefc74d8454841c730f3a0b165e0a\": rpc error: code = NotFound desc = could not find container \"98b8e724c676bfdd4aed90fda3dbf32b5dccefc74d8454841c730f3a0b165e0a\": container with ID starting with 98b8e724c676bfdd4aed90fda3dbf32b5dccefc74d8454841c730f3a0b165e0a not found: ID does not exist"
Sep 11 18:03:17.052742 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:17.052705    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28b282f7-eeab-422d-b460-0c927e390749" path="/var/lib/kubelet/pods/28b282f7-eeab-422d-b460-0c927e390749/volumes"
Sep 11 18:03:19.632213 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:19.632177    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"]
Sep 11 18:03:19.636617 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:19.636592    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-fb8c5d/test-model-fb8c5d-kserve-844f86dd9c-kjmtk"]
Sep 11 18:03:21.053020 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:21.052982    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34d571d9-7de8-45f1-8779-e60153247d9d" path="/var/lib/kubelet/pods/34d571d9-7de8-45f1-8779-e60153247d9d/volumes"
Sep 11 18:03:23.104373 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:23.104319    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:03:35.075991 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:35.075951    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"]
Sep 11 18:03:35.076507 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:35.076204    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5" podUID="b8b217c2-e087-40d0-878c-1224cd1675e7" containerName="maas-api" containerID="cri-o://f9fdd70f8ba5e332a3259d10befe1a258657c018db5ec46d63fcd1aee5bf2cfb" gracePeriod=30
Sep 11 18:03:35.335584 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:35.335504    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"
Sep 11 18:03:35.444561 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:35.444525    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8b217c2-e087-40d0-878c-1224cd1675e7-metrics-certs\") pod \"b8b217c2-e087-40d0-878c-1224cd1675e7\" (UID: \"b8b217c2-e087-40d0-878c-1224cd1675e7\") "
Sep 11 18:03:35.444719 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:35.444611    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xn4c9\" (UniqueName: \"kubernetes.io/projected/b8b217c2-e087-40d0-878c-1224cd1675e7-kube-api-access-xn4c9\") pod \"b8b217c2-e087-40d0-878c-1224cd1675e7\" (UID: \"b8b217c2-e087-40d0-878c-1224cd1675e7\") "
Sep 11 18:03:35.444719 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:35.444658    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b217c2-e087-40d0-878c-1224cd1675e7-maas-api-tls\") pod \"b8b217c2-e087-40d0-878c-1224cd1675e7\" (UID: \"b8b217c2-e087-40d0-878c-1224cd1675e7\") "
Sep 11 18:03:35.444719 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:35.444699    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b217c2-e087-40d0-878c-1224cd1675e7-openshift-service-ca\") pod \"b8b217c2-e087-40d0-878c-1224cd1675e7\" (UID: \"b8b217c2-e087-40d0-878c-1224cd1675e7\") "
Sep 11 18:03:35.445118 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:35.445090    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b217c2-e087-40d0-878c-1224cd1675e7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b8b217c2-e087-40d0-878c-1224cd1675e7" (UID: "b8b217c2-e087-40d0-878c-1224cd1675e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:03:35.446781 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:35.446746    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b217c2-e087-40d0-878c-1224cd1675e7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b8b217c2-e087-40d0-878c-1224cd1675e7" (UID: "b8b217c2-e087-40d0-878c-1224cd1675e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:03:35.446781 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:35.446765    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b217c2-e087-40d0-878c-1224cd1675e7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b8b217c2-e087-40d0-878c-1224cd1675e7" (UID: "b8b217c2-e087-40d0-878c-1224cd1675e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:03:35.446926 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:35.446860    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b217c2-e087-40d0-878c-1224cd1675e7-kube-api-access-xn4c9" (OuterVolumeSpecName: "kube-api-access-xn4c9") pod "b8b217c2-e087-40d0-878c-1224cd1675e7" (UID: "b8b217c2-e087-40d0-878c-1224cd1675e7"). InnerVolumeSpecName "kube-api-access-xn4c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:03:35.546005 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:35.545966    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b217c2-e087-40d0-878c-1224cd1675e7-maas-api-tls\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:03:35.546005 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:35.546000    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b217c2-e087-40d0-878c-1224cd1675e7-openshift-service-ca\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:03:35.546005 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:35.546010    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8b217c2-e087-40d0-878c-1224cd1675e7-metrics-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:03:35.546239 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:35.546019    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn4c9\" (UniqueName: \"kubernetes.io/projected/b8b217c2-e087-40d0-878c-1224cd1675e7-kube-api-access-xn4c9\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:03:35.803147 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:35.803106    2559 generic.go:356] "Generic (PLEG): container finished" podID="b8b217c2-e087-40d0-878c-1224cd1675e7" containerID="f9fdd70f8ba5e332a3259d10befe1a258657c018db5ec46d63fcd1aee5bf2cfb" exitCode=0
Sep 11 18:03:35.803416 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:35.803156    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"
Sep 11 18:03:35.803416 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:35.803184    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5" event={"ID":"b8b217c2-e087-40d0-878c-1224cd1675e7","Type":"ContainerDied","Data":"f9fdd70f8ba5e332a3259d10befe1a258657c018db5ec46d63fcd1aee5bf2cfb"}
Sep 11 18:03:35.803416 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:35.803241    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5" event={"ID":"b8b217c2-e087-40d0-878c-1224cd1675e7","Type":"ContainerDied","Data":"d6ea9041d06e335a172f00fc5a41bb48821adf6efb2be22baca47bbd927115c1"}
Sep 11 18:03:35.803416 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:35.803263    2559 scope.go:117] "RemoveContainer" containerID="f9fdd70f8ba5e332a3259d10befe1a258657c018db5ec46d63fcd1aee5bf2cfb"
Sep 11 18:03:35.812693 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:35.812675    2559 scope.go:117] "RemoveContainer" containerID="f9fdd70f8ba5e332a3259d10befe1a258657c018db5ec46d63fcd1aee5bf2cfb"
Sep 11 18:03:35.812969 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:03:35.812947    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9fdd70f8ba5e332a3259d10befe1a258657c018db5ec46d63fcd1aee5bf2cfb\": container with ID starting with f9fdd70f8ba5e332a3259d10befe1a258657c018db5ec46d63fcd1aee5bf2cfb not found: ID does not exist" containerID="f9fdd70f8ba5e332a3259d10befe1a258657c018db5ec46d63fcd1aee5bf2cfb"
Sep 11 18:03:35.813042 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:35.812976    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9fdd70f8ba5e332a3259d10befe1a258657c018db5ec46d63fcd1aee5bf2cfb"} err="failed to get container status \"f9fdd70f8ba5e332a3259d10befe1a258657c018db5ec46d63fcd1aee5bf2cfb\": rpc error: code = NotFound desc = could not find container \"f9fdd70f8ba5e332a3259d10befe1a258657c018db5ec46d63fcd1aee5bf2cfb\": container with ID starting with f9fdd70f8ba5e332a3259d10befe1a258657c018db5ec46d63fcd1aee5bf2cfb not found: ID does not exist"
Sep 11 18:03:35.844927 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:35.844898    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"]
Sep 11 18:03:35.849716 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:35.849688    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fb8c5d-7fdb496447-7wwh5"]
Sep 11 18:03:37.052952 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:37.052915    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8b217c2-e087-40d0-878c-1224cd1675e7" path="/var/lib/kubelet/pods/b8b217c2-e087-40d0-878c-1224cd1675e7/volumes"
Sep 11 18:03:38.077372 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:38.077317    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:03:48.762924 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:48.762887    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:03:51.707905 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.707873    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h"]
Sep 11 18:03:51.708305 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.708203    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28b282f7-eeab-422d-b460-0c927e390749" containerName="main"
Sep 11 18:03:51.708305 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.708214    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="28b282f7-eeab-422d-b460-0c927e390749" containerName="main"
Sep 11 18:03:51.708305 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.708227    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b217c2-e087-40d0-878c-1224cd1675e7" containerName="maas-api"
Sep 11 18:03:51.708305 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.708233    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b217c2-e087-40d0-878c-1224cd1675e7" containerName="maas-api"
Sep 11 18:03:51.708305 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.708243    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34d571d9-7de8-45f1-8779-e60153247d9d" containerName="main"
Sep 11 18:03:51.708305 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.708249    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="34d571d9-7de8-45f1-8779-e60153247d9d" containerName="main"
Sep 11 18:03:51.708305 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.708260    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf" containerName="main"
Sep 11 18:03:51.708305 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.708265    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf" containerName="main"
Sep 11 18:03:51.708657 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.708334    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="28b282f7-eeab-422d-b460-0c927e390749" containerName="main"
Sep 11 18:03:51.708657 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.708342    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="34d571d9-7de8-45f1-8779-e60153247d9d" containerName="main"
Sep 11 18:03:51.708657 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.708362    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3f40e3b-f9e8-499e-aff1-b3e5d7027bbf" containerName="main"
Sep 11 18:03:51.708657 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.708369    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b217c2-e087-40d0-878c-1224cd1675e7" containerName="maas-api"
Sep 11 18:03:51.712987 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.712963    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h"
Sep 11 18:03:51.743247 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.743206    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 18:03:51.743519 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.743503    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:03:51.744164 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.744140    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-386935-serving-cert\""
Sep 11 18:03:51.744341 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.744147    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 18:03:51.747876 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.747852    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cdcpv\""
Sep 11 18:03:51.748687 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.748662    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h"]
Sep 11 18:03:51.802990 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.802951    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cda3d66-3ee6-468f-9fd4-fab720ed56c2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h\" (UID: \"4cda3d66-3ee6-468f-9fd4-fab720ed56c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h"
Sep 11 18:03:51.803186 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.803025    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cda3d66-3ee6-468f-9fd4-fab720ed56c2-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h\" (UID: \"4cda3d66-3ee6-468f-9fd4-fab720ed56c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h"
Sep 11 18:03:51.803186 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.803117    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cda3d66-3ee6-468f-9fd4-fab720ed56c2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h\" (UID: \"4cda3d66-3ee6-468f-9fd4-fab720ed56c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h"
Sep 11 18:03:51.803186 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.803147    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-654fr\" (UniqueName: \"kubernetes.io/projected/4cda3d66-3ee6-468f-9fd4-fab720ed56c2-kube-api-access-654fr\") pod \"maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h\" (UID: \"4cda3d66-3ee6-468f-9fd4-fab720ed56c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h"
Sep 11 18:03:51.904313 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.904270    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cda3d66-3ee6-468f-9fd4-fab720ed56c2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h\" (UID: \"4cda3d66-3ee6-468f-9fd4-fab720ed56c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h"
Sep 11 18:03:51.904508 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.904329    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cda3d66-3ee6-468f-9fd4-fab720ed56c2-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h\" (UID: \"4cda3d66-3ee6-468f-9fd4-fab720ed56c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h"
Sep 11 18:03:51.904508 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.904394    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cda3d66-3ee6-468f-9fd4-fab720ed56c2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h\" (UID: \"4cda3d66-3ee6-468f-9fd4-fab720ed56c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h"
Sep 11 18:03:51.904508 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.904413    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-654fr\" (UniqueName: \"kubernetes.io/projected/4cda3d66-3ee6-468f-9fd4-fab720ed56c2-kube-api-access-654fr\") pod \"maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h\" (UID: \"4cda3d66-3ee6-468f-9fd4-fab720ed56c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h"
Sep 11 18:03:51.905118 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.905097    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cda3d66-3ee6-468f-9fd4-fab720ed56c2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h\" (UID: \"4cda3d66-3ee6-468f-9fd4-fab720ed56c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h"
Sep 11 18:03:51.906824 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.906796    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cda3d66-3ee6-468f-9fd4-fab720ed56c2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h\" (UID: \"4cda3d66-3ee6-468f-9fd4-fab720ed56c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h"
Sep 11 18:03:51.906930 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.906837    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cda3d66-3ee6-468f-9fd4-fab720ed56c2-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h\" (UID: \"4cda3d66-3ee6-468f-9fd4-fab720ed56c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h"
Sep 11 18:03:51.930291 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:51.930257    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-654fr\" (UniqueName: \"kubernetes.io/projected/4cda3d66-3ee6-468f-9fd4-fab720ed56c2-kube-api-access-654fr\") pod \"maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h\" (UID: \"4cda3d66-3ee6-468f-9fd4-fab720ed56c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h"
Sep 11 18:03:52.023510 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:52.023407    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h"
Sep 11 18:03:52.339791 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:03:52.339760    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cda3d66_3ee6_468f_9fd4_fab720ed56c2.slice/crio-8b16e4db41e006480535443e5062b92a42a7f4a6c37fd6f32e9e2951762932d8 WatchSource:0}: Error finding container 8b16e4db41e006480535443e5062b92a42a7f4a6c37fd6f32e9e2951762932d8: Status 404 returned error can't find the container with id 8b16e4db41e006480535443e5062b92a42a7f4a6c37fd6f32e9e2951762932d8
Sep 11 18:03:52.341202 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:52.341180    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:03:52.341302 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:52.341282    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h"]
Sep 11 18:03:52.898920 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:52.898869    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h" event={"ID":"4cda3d66-3ee6-468f-9fd4-fab720ed56c2","Type":"ContainerStarted","Data":"05eeb622f61f136742f0fe213784b4dc17ad8df90accd513b658947f1175fadf"}
Sep 11 18:03:52.899390 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:52.898957    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h" event={"ID":"4cda3d66-3ee6-468f-9fd4-fab720ed56c2","Type":"ContainerStarted","Data":"8b16e4db41e006480535443e5062b92a42a7f4a6c37fd6f32e9e2951762932d8"}
Sep 11 18:03:52.899390 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:52.899084    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h"
Sep 11 18:03:52.962971 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:52.962858    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h" podStartSLOduration=1.649003713 podStartE2EDuration="1.962841246s" totalImagesPullingTime="313.837533ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:03:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:03:52.959383541 +0000 UTC m=+1138.404484113" watchObservedRunningTime="2026-09-11 18:03:52.962841246 +0000 UTC m=+1138.407941818"
Sep 11 18:03:58.911005 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:03:58.910972    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h"
Sep 11 18:04:01.672550 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:01.672514    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:04:18.014432 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:18.014336    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz"]
Sep 11 18:04:18.017984 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:18.017965    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz"
Sep 11 18:04:18.021810 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:18.021789    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-f95803-serving-cert\""
Sep 11 18:04:18.131102 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:18.131061    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz"]
Sep 11 18:04:18.154379 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:18.151219    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5jdq\" (UniqueName: \"kubernetes.io/projected/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09-kube-api-access-c5jdq\") pod \"maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz\" (UID: \"1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz"
Sep 11 18:04:18.154379 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:18.151333    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz\" (UID: \"1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz"
Sep 11 18:04:18.154379 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:18.151428    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz\" (UID: \"1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz"
Sep 11 18:04:18.154379 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:18.151483    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz\" (UID: \"1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz"
Sep 11 18:04:18.252519 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:18.252476    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5jdq\" (UniqueName: \"kubernetes.io/projected/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09-kube-api-access-c5jdq\") pod \"maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz\" (UID: \"1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz"
Sep 11 18:04:18.252725 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:18.252554    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz\" (UID: \"1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz"
Sep 11 18:04:18.252725 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:18.252589    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz\" (UID: \"1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz"
Sep 11 18:04:18.252725 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:18.252625    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz\" (UID: \"1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz"
Sep 11 18:04:18.253205 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:18.253179    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz\" (UID: \"1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz"
Sep 11 18:04:18.255084 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:18.255050    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz\" (UID: \"1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz"
Sep 11 18:04:18.255190 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:18.255114    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz\" (UID: \"1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz"
Sep 11 18:04:18.269380 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:18.269295    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5jdq\" (UniqueName: \"kubernetes.io/projected/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09-kube-api-access-c5jdq\") pod \"maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz\" (UID: \"1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz"
Sep 11 18:04:18.328329 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:18.328285    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz"
Sep 11 18:04:18.627641 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:18.627608    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz"]
Sep 11 18:04:18.628125 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:04:18.628082    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8bd8df_3927_4ff7_ae3e_e1d9f8604e09.slice/crio-f06faf337440564fd18cddc6b60d3abd6bc8057f247b876dee1ab5fa6d962c02 WatchSource:0}: Error finding container f06faf337440564fd18cddc6b60d3abd6bc8057f247b876dee1ab5fa6d962c02: Status 404 returned error can't find the container with id f06faf337440564fd18cddc6b60d3abd6bc8057f247b876dee1ab5fa6d962c02
Sep 11 18:04:19.035163 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:19.035135    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz" event={"ID":"1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09","Type":"ContainerStarted","Data":"f06faf337440564fd18cddc6b60d3abd6bc8057f247b876dee1ab5fa6d962c02"}
Sep 11 18:04:20.043075 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:20.043032    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz" event={"ID":"1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09","Type":"ContainerStarted","Data":"5f4d53ff4699c833ec6318ac3fcf68b4ae9afb3ac77203998ff28f2ab01b22c8"}
Sep 11 18:04:20.043513 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:20.043103    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz"
Sep 11 18:04:20.072112 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:20.072032    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz" podStartSLOduration=2.669181068 podStartE2EDuration="3.072012147s" totalImagesPullingTime="402.831079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:04:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:04:20.071154246 +0000 UTC m=+1165.516254817" watchObservedRunningTime="2026-09-11 18:04:20.072012147 +0000 UTC m=+1165.517112718"
Sep 11 18:04:21.826979 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:21.826938    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"]
Sep 11 18:04:21.830709 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:21.830690    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"
Sep 11 18:04:21.835202 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:21.835171    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-3ebea5-serving-cert\""
Sep 11 18:04:21.865666 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:21.865631    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"]
Sep 11 18:04:21.890009 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:21.889965    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1768d910-3783-43aa-8b6c-1857bdc4b3c8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m\" (UID: \"1768d910-3783-43aa-8b6c-1857bdc4b3c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"
Sep 11 18:04:21.890210 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:21.890019    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srgnl\" (UniqueName: \"kubernetes.io/projected/1768d910-3783-43aa-8b6c-1857bdc4b3c8-kube-api-access-srgnl\") pod \"maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m\" (UID: \"1768d910-3783-43aa-8b6c-1857bdc4b3c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"
Sep 11 18:04:21.890210 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:21.890134    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1768d910-3783-43aa-8b6c-1857bdc4b3c8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m\" (UID: \"1768d910-3783-43aa-8b6c-1857bdc4b3c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"
Sep 11 18:04:21.890321 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:21.890213    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1768d910-3783-43aa-8b6c-1857bdc4b3c8-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m\" (UID: \"1768d910-3783-43aa-8b6c-1857bdc4b3c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"
Sep 11 18:04:21.991020 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:21.990979    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1768d910-3783-43aa-8b6c-1857bdc4b3c8-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m\" (UID: \"1768d910-3783-43aa-8b6c-1857bdc4b3c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"
Sep 11 18:04:21.991269 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:21.991077    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1768d910-3783-43aa-8b6c-1857bdc4b3c8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m\" (UID: \"1768d910-3783-43aa-8b6c-1857bdc4b3c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"
Sep 11 18:04:21.991269 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:21.991111    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srgnl\" (UniqueName: \"kubernetes.io/projected/1768d910-3783-43aa-8b6c-1857bdc4b3c8-kube-api-access-srgnl\") pod \"maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m\" (UID: \"1768d910-3783-43aa-8b6c-1857bdc4b3c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"
Sep 11 18:04:21.991269 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:21.991173    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1768d910-3783-43aa-8b6c-1857bdc4b3c8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m\" (UID: \"1768d910-3783-43aa-8b6c-1857bdc4b3c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"
Sep 11 18:04:21.991491 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:04:21.991315    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-serving-cert: secret "maas-api-e2e-shared-b-w3-3ebea5-serving-cert" not found
Sep 11 18:04:21.991491 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:04:21.991427    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1768d910-3783-43aa-8b6c-1857bdc4b3c8-maas-api-tls podName:1768d910-3783-43aa-8b6c-1857bdc4b3c8 nodeName:}" failed. No retries permitted until 2026-09-11 18:04:22.491405885 +0000 UTC m=+1167.936506440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1768d910-3783-43aa-8b6c-1857bdc4b3c8-maas-api-tls") pod "maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m" (UID: "1768d910-3783-43aa-8b6c-1857bdc4b3c8") : secret "maas-api-e2e-shared-b-w3-3ebea5-serving-cert" not found
Sep 11 18:04:21.991790 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:21.991766    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1768d910-3783-43aa-8b6c-1857bdc4b3c8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m\" (UID: \"1768d910-3783-43aa-8b6c-1857bdc4b3c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"
Sep 11 18:04:21.993538 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:21.993518    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1768d910-3783-43aa-8b6c-1857bdc4b3c8-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m\" (UID: \"1768d910-3783-43aa-8b6c-1857bdc4b3c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"
Sep 11 18:04:22.010332 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:22.010304    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srgnl\" (UniqueName: \"kubernetes.io/projected/1768d910-3783-43aa-8b6c-1857bdc4b3c8-kube-api-access-srgnl\") pod \"maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m\" (UID: \"1768d910-3783-43aa-8b6c-1857bdc4b3c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"
Sep 11 18:04:22.411825 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:22.411791    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:04:22.496548 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:22.496504    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1768d910-3783-43aa-8b6c-1857bdc4b3c8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m\" (UID: \"1768d910-3783-43aa-8b6c-1857bdc4b3c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"
Sep 11 18:04:22.499235 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:22.499207    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1768d910-3783-43aa-8b6c-1857bdc4b3c8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m\" (UID: \"1768d910-3783-43aa-8b6c-1857bdc4b3c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"
Sep 11 18:04:22.741130 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:22.741030    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"
Sep 11 18:04:23.073448 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:23.073421    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"]
Sep 11 18:04:23.075881 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:04:23.075854    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1768d910_3783_43aa_8b6c_1857bdc4b3c8.slice/crio-1a06fc94311d91f40e965ebfee4236e5fc557fe512f129942f7252ee50446517 WatchSource:0}: Error finding container 1a06fc94311d91f40e965ebfee4236e5fc557fe512f129942f7252ee50446517: Status 404 returned error can't find the container with id 1a06fc94311d91f40e965ebfee4236e5fc557fe512f129942f7252ee50446517
Sep 11 18:04:24.066553 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:24.066452    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m" event={"ID":"1768d910-3783-43aa-8b6c-1857bdc4b3c8","Type":"ContainerStarted","Data":"00ec66916f062fa3ba9cdc3fa04613a16a444a20b059cc89d1965936b388e450"}
Sep 11 18:04:24.066553 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:24.066511    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m" event={"ID":"1768d910-3783-43aa-8b6c-1857bdc4b3c8","Type":"ContainerStarted","Data":"1a06fc94311d91f40e965ebfee4236e5fc557fe512f129942f7252ee50446517"}
Sep 11 18:04:24.066812 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:24.066550    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"
Sep 11 18:04:24.144774 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:24.144716    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m" podStartSLOduration=2.526809031 podStartE2EDuration="3.144699467s" totalImagesPullingTime="617.890436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:04:24.138013618 +0000 UTC m=+1169.583114189" watchObservedRunningTime="2026-09-11 18:04:24.144699467 +0000 UTC m=+1169.589800094"
Sep 11 18:04:26.054364 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:26.054316    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz"
Sep 11 18:04:29.639216 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:29.639177    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7862-2f4745"]
Sep 11 18:04:29.643525 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:29.643499    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7862-2f4745"
Sep 11 18:04:29.647118 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:29.647093    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nbc6p\""
Sep 11 18:04:29.652906 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:29.652880    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7862-2f4745"]
Sep 11 18:04:29.765908 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:29.765867    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp999\" (UniqueName: \"kubernetes.io/projected/e0cb53f7-2d7f-423a-ad1c-908410779756-kube-api-access-rp999\") pod \"test-curl-7862-2f4745\" (UID: \"e0cb53f7-2d7f-423a-ad1c-908410779756\") " pod="opendatahub/test-curl-7862-2f4745"
Sep 11 18:04:29.866986 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:29.866949    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp999\" (UniqueName: \"kubernetes.io/projected/e0cb53f7-2d7f-423a-ad1c-908410779756-kube-api-access-rp999\") pod \"test-curl-7862-2f4745\" (UID: \"e0cb53f7-2d7f-423a-ad1c-908410779756\") " pod="opendatahub/test-curl-7862-2f4745"
Sep 11 18:04:29.877992 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:29.877956    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp999\" (UniqueName: \"kubernetes.io/projected/e0cb53f7-2d7f-423a-ad1c-908410779756-kube-api-access-rp999\") pod \"test-curl-7862-2f4745\" (UID: \"e0cb53f7-2d7f-423a-ad1c-908410779756\") " pod="opendatahub/test-curl-7862-2f4745"
Sep 11 18:04:29.954425 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:29.954290    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7862-2f4745"
Sep 11 18:04:30.077643 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:30.077614    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"
Sep 11 18:04:30.249089 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:30.249056    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7862-2f4745"]
Sep 11 18:04:30.249541 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:04:30.249516    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0cb53f7_2d7f_423a_ad1c_908410779756.slice/crio-271f3e3b433de81fbe1e5a92d6669571edb95d64b4bd2f0090e0669b4587ee35 WatchSource:0}: Error finding container 271f3e3b433de81fbe1e5a92d6669571edb95d64b4bd2f0090e0669b4587ee35: Status 404 returned error can't find the container with id 271f3e3b433de81fbe1e5a92d6669571edb95d64b4bd2f0090e0669b4587ee35
Sep 11 18:04:31.110861 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:31.110817    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7862-2f4745" event={"ID":"e0cb53f7-2d7f-423a-ad1c-908410779756","Type":"ContainerStarted","Data":"c2ad53b29e6254456742898d16fdfe86901b16c810f712e65e6c4383d16d72c5"}
Sep 11 18:04:31.111291 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:31.110871    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7862-2f4745" event={"ID":"e0cb53f7-2d7f-423a-ad1c-908410779756","Type":"ContainerStarted","Data":"271f3e3b433de81fbe1e5a92d6669571edb95d64b4bd2f0090e0669b4587ee35"}
Sep 11 18:04:31.130111 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:31.130044    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7862-2f4745" podStartSLOduration=2.130023222 podStartE2EDuration="2.130023222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:04:31.128099614 +0000 UTC m=+1176.573200186" watchObservedRunningTime="2026-09-11 18:04:31.130023222 +0000 UTC m=+1176.575123796"
Sep 11 18:04:31.866252 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:31.866165    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7862-2f4745"]
Sep 11 18:04:31.877649 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:31.877613    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7862-2f4745"]
Sep 11 18:04:32.004585 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:32.004552    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7862-a67557"]
Sep 11 18:04:32.004937 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:32.004925    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0cb53f7-2d7f-423a-ad1c-908410779756" containerName="test-curl-7862-2f4745"
Sep 11 18:04:32.004982 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:32.004939    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0cb53f7-2d7f-423a-ad1c-908410779756" containerName="test-curl-7862-2f4745"
Sep 11 18:04:32.005018 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:32.005011    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0cb53f7-2d7f-423a-ad1c-908410779756" containerName="test-curl-7862-2f4745"
Sep 11 18:04:32.008278 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:32.008262    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7862-a67557"
Sep 11 18:04:32.021633 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:32.021602    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7862-a67557"]
Sep 11 18:04:32.116312 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:04:32.116231    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7862-2f4745" container="test-curl-7862-2f4745"
Sep 11 18:04:32.188801 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:32.188758    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65hjv\" (UniqueName: \"kubernetes.io/projected/937920d8-8daa-472f-a499-accce6cbbee9-kube-api-access-65hjv\") pod \"test-curl-7862-a67557\" (UID: \"937920d8-8daa-472f-a499-accce6cbbee9\") " pod="opendatahub/test-curl-7862-a67557"
Sep 11 18:04:32.290070 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:32.290030    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65hjv\" (UniqueName: \"kubernetes.io/projected/937920d8-8daa-472f-a499-accce6cbbee9-kube-api-access-65hjv\") pod \"test-curl-7862-a67557\" (UID: \"937920d8-8daa-472f-a499-accce6cbbee9\") " pod="opendatahub/test-curl-7862-a67557"
Sep 11 18:04:32.310725 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:32.310695    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65hjv\" (UniqueName: \"kubernetes.io/projected/937920d8-8daa-472f-a499-accce6cbbee9-kube-api-access-65hjv\") pod \"test-curl-7862-a67557\" (UID: \"937920d8-8daa-472f-a499-accce6cbbee9\") " pod="opendatahub/test-curl-7862-a67557"
Sep 11 18:04:32.317443 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:32.317417    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7862-a67557"
Sep 11 18:04:32.602639 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:32.602609    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7862-a67557"]
Sep 11 18:04:32.604066 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:04:32.604039    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod937920d8_8daa_472f_a499_accce6cbbee9.slice/crio-ad76db7891cd30cbc09b76e857e4b6c7be70f32374f8a4efe41d2cc81f513476 WatchSource:0}: Error finding container ad76db7891cd30cbc09b76e857e4b6c7be70f32374f8a4efe41d2cc81f513476: Status 404 returned error can't find the container with id ad76db7891cd30cbc09b76e857e4b6c7be70f32374f8a4efe41d2cc81f513476
Sep 11 18:04:33.124016 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:33.123978    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7862-a67557" event={"ID":"937920d8-8daa-472f-a499-accce6cbbee9","Type":"ContainerStarted","Data":"fd422db58f1b61f489f9ea4417f62be24d3d10124ba940a106c7f5516a6f0dfd"}
Sep 11 18:04:33.124470 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:33.124025    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7862-a67557" event={"ID":"937920d8-8daa-472f-a499-accce6cbbee9","Type":"ContainerStarted","Data":"ad76db7891cd30cbc09b76e857e4b6c7be70f32374f8a4efe41d2cc81f513476"}
Sep 11 18:04:33.124470 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:33.124032    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7862-2f4745" podUID="e0cb53f7-2d7f-423a-ad1c-908410779756" containerName="test-curl-7862-2f4745" containerID="cri-o://c2ad53b29e6254456742898d16fdfe86901b16c810f712e65e6c4383d16d72c5" gracePeriod=2
Sep 11 18:04:33.170269 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:33.170215    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7862-a67557" podStartSLOduration=2.170197344 podStartE2EDuration="2.170197344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:04:33.168585829 +0000 UTC m=+1178.613686411" watchObservedRunningTime="2026-09-11 18:04:33.170197344 +0000 UTC m=+1178.615297914"
Sep 11 18:04:35.119269 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:35.119179    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7862-a67557"]
Sep 11 18:04:35.122550 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:35.122520    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7862-a67557"]
Sep 11 18:04:35.136230 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:35.136079    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7862-a67557" podUID="937920d8-8daa-472f-a499-accce6cbbee9" containerName="test-curl-7862-a67557" containerID="cri-o://fd422db58f1b61f489f9ea4417f62be24d3d10124ba940a106c7f5516a6f0dfd" gracePeriod=2
Sep 11 18:04:35.140412 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:04:35.140382    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7862-a67557\" is forbidden: User \"system:node:ip-10-0-138-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-96.ec2.internal' and this object" podUID="937920d8-8daa-472f-a499-accce6cbbee9" pod="opendatahub/test-curl-7862-a67557"
Sep 11 18:04:35.254482 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:35.254442    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7862-9fff8c"]
Sep 11 18:04:35.254880 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:35.254861    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="937920d8-8daa-472f-a499-accce6cbbee9" containerName="test-curl-7862-a67557"
Sep 11 18:04:35.254976 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:35.254883    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="937920d8-8daa-472f-a499-accce6cbbee9" containerName="test-curl-7862-a67557"
Sep 11 18:04:35.255028 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:35.255005    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="937920d8-8daa-472f-a499-accce6cbbee9" containerName="test-curl-7862-a67557"
Sep 11 18:04:35.258000 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:35.257980    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7862-9fff8c"
Sep 11 18:04:35.273129 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:35.273103    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7862-9fff8c"]
Sep 11 18:04:35.294101 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:04:35.294066    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7862-a67557\" is forbidden: User \"system:node:ip-10-0-138-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-96.ec2.internal' and this object" podUID="937920d8-8daa-472f-a499-accce6cbbee9" pod="opendatahub/test-curl-7862-a67557"
Sep 11 18:04:35.309954 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:35.309928    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7862-2f4745"
Sep 11 18:04:35.312091 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:04:35.312063    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7862-a67557\" is forbidden: User \"system:node:ip-10-0-138-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-96.ec2.internal' and this object" podUID="937920d8-8daa-472f-a499-accce6cbbee9" pod="opendatahub/test-curl-7862-a67557"
Sep 11 18:04:35.418881 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:35.418846    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp999\" (UniqueName: \"kubernetes.io/projected/e0cb53f7-2d7f-423a-ad1c-908410779756-kube-api-access-rp999\") pod \"e0cb53f7-2d7f-423a-ad1c-908410779756\" (UID: \"e0cb53f7-2d7f-423a-ad1c-908410779756\") "
Sep 11 18:04:35.419063 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:35.419050    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47ldt\" (UniqueName: \"kubernetes.io/projected/b59929f6-cb0e-434c-9cc5-948785c84eaa-kube-api-access-47ldt\") pod \"test-curl-7862-9fff8c\" (UID: \"b59929f6-cb0e-434c-9cc5-948785c84eaa\") " pod="opendatahub/test-curl-7862-9fff8c"
Sep 11 18:04:35.420970 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:35.420942    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0cb53f7-2d7f-423a-ad1c-908410779756-kube-api-access-rp999" (OuterVolumeSpecName: "kube-api-access-rp999") pod "e0cb53f7-2d7f-423a-ad1c-908410779756" (UID: "e0cb53f7-2d7f-423a-ad1c-908410779756"). InnerVolumeSpecName "kube-api-access-rp999". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:04:35.519981 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:35.519944    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47ldt\" (UniqueName: \"kubernetes.io/projected/b59929f6-cb0e-434c-9cc5-948785c84eaa-kube-api-access-47ldt\") pod \"test-curl-7862-9fff8c\" (UID: \"b59929f6-cb0e-434c-9cc5-948785c84eaa\") " pod="opendatahub/test-curl-7862-9fff8c"
Sep 11 18:04:35.520150 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:35.520011    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp999\" (UniqueName: \"kubernetes.io/projected/e0cb53f7-2d7f-423a-ad1c-908410779756-kube-api-access-rp999\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:04:35.529543 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:35.529516    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47ldt\" (UniqueName: \"kubernetes.io/projected/b59929f6-cb0e-434c-9cc5-948785c84eaa-kube-api-access-47ldt\") pod \"test-curl-7862-9fff8c\" (UID: \"b59929f6-cb0e-434c-9cc5-948785c84eaa\") " pod="opendatahub/test-curl-7862-9fff8c"
Sep 11 18:04:35.567568 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:35.567535    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7862-9fff8c"
Sep 11 18:04:35.875834 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:35.875740    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7862-9fff8c"]
Sep 11 18:04:35.888295 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:04:35.888267    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb59929f6_cb0e_434c_9cc5_948785c84eaa.slice/crio-78f841feb6e6cb289ceb44bf0872a84a3f75a4d20ff1a2ab45a81ba081edba7e WatchSource:0}: Error finding container 78f841feb6e6cb289ceb44bf0872a84a3f75a4d20ff1a2ab45a81ba081edba7e: Status 404 returned error can't find the container with id 78f841feb6e6cb289ceb44bf0872a84a3f75a4d20ff1a2ab45a81ba081edba7e
Sep 11 18:04:36.143059 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:36.143020    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7862-9fff8c" event={"ID":"b59929f6-cb0e-434c-9cc5-948785c84eaa","Type":"ContainerStarted","Data":"eeb4936208a306b7bb06e1148313bc962f12e64cbc5c2d4a61b987ca99595a34"}
Sep 11 18:04:36.143540 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:36.143073    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7862-9fff8c" event={"ID":"b59929f6-cb0e-434c-9cc5-948785c84eaa","Type":"ContainerStarted","Data":"78f841feb6e6cb289ceb44bf0872a84a3f75a4d20ff1a2ab45a81ba081edba7e"}
Sep 11 18:04:36.144642 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:36.144613    2559 generic.go:356] "Generic (PLEG): container finished" podID="e0cb53f7-2d7f-423a-ad1c-908410779756" containerID="c2ad53b29e6254456742898d16fdfe86901b16c810f712e65e6c4383d16d72c5" exitCode=137
Sep 11 18:04:36.144780 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:36.144656    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7862-2f4745"
Sep 11 18:04:36.144780 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:36.144703    2559 scope.go:117] "RemoveContainer" containerID="c2ad53b29e6254456742898d16fdfe86901b16c810f712e65e6c4383d16d72c5"
Sep 11 18:04:36.154300 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:36.154285    2559 scope.go:117] "RemoveContainer" containerID="c2ad53b29e6254456742898d16fdfe86901b16c810f712e65e6c4383d16d72c5"
Sep 11 18:04:36.154594 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:04:36.154574    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2ad53b29e6254456742898d16fdfe86901b16c810f712e65e6c4383d16d72c5\": container with ID starting with c2ad53b29e6254456742898d16fdfe86901b16c810f712e65e6c4383d16d72c5 not found: ID does not exist" containerID="c2ad53b29e6254456742898d16fdfe86901b16c810f712e65e6c4383d16d72c5"
Sep 11 18:04:36.154638 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:36.154603    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2ad53b29e6254456742898d16fdfe86901b16c810f712e65e6c4383d16d72c5"} err="failed to get container status \"c2ad53b29e6254456742898d16fdfe86901b16c810f712e65e6c4383d16d72c5\": rpc error: code = NotFound desc = could not find container \"c2ad53b29e6254456742898d16fdfe86901b16c810f712e65e6c4383d16d72c5\": container with ID starting with c2ad53b29e6254456742898d16fdfe86901b16c810f712e65e6c4383d16d72c5 not found: ID does not exist"
Sep 11 18:04:36.161407 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:36.161335    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7862-9fff8c" podStartSLOduration=1.161323056 podStartE2EDuration="1.161323056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:04:36.160835795 +0000 UTC m=+1181.605936365" watchObservedRunningTime="2026-09-11 18:04:36.161323056 +0000 UTC m=+1181.606423627"
Sep 11 18:04:36.163163 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:04:36.163136    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7862-a67557\" is forbidden: User \"system:node:ip-10-0-138-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-96.ec2.internal' and this object" podUID="937920d8-8daa-472f-a499-accce6cbbee9" pod="opendatahub/test-curl-7862-a67557"
Sep 11 18:04:36.164942 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:04:36.164918    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7862-a67557\" is forbidden: User \"system:node:ip-10-0-138-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-96.ec2.internal' and this object" podUID="937920d8-8daa-472f-a499-accce6cbbee9" pod="opendatahub/test-curl-7862-a67557"
Sep 11 18:04:36.166969 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:04:36.166949    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7862-2f4745\" is forbidden: User \"system:node:ip-10-0-138-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-96.ec2.internal' and this object" podUID="e0cb53f7-2d7f-423a-ad1c-908410779756" pod="opendatahub/test-curl-7862-2f4745"
Sep 11 18:04:36.879941 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:36.879900    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7862-9fff8c"]
Sep 11 18:04:36.886695 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:36.886657    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7862-9fff8c"]
Sep 11 18:04:37.014545 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:37.014466    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7862-5d0f8b"]
Sep 11 18:04:37.014845 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:37.014832    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b59929f6-cb0e-434c-9cc5-948785c84eaa" containerName="test-curl-7862-9fff8c"
Sep 11 18:04:37.014888 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:37.014846    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b59929f6-cb0e-434c-9cc5-948785c84eaa" containerName="test-curl-7862-9fff8c"
Sep 11 18:04:37.014936 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:37.014926    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b59929f6-cb0e-434c-9cc5-948785c84eaa" containerName="test-curl-7862-9fff8c"
Sep 11 18:04:37.019259 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:37.019232    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7862-5d0f8b"
Sep 11 18:04:37.027217 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:37.027188    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7862-5d0f8b"]
Sep 11 18:04:37.040482 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:04:37.040451    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7862-a67557\" is forbidden: User \"system:node:ip-10-0-138-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-96.ec2.internal' and this object" podUID="937920d8-8daa-472f-a499-accce6cbbee9" pod="opendatahub/test-curl-7862-a67557"
Sep 11 18:04:37.052481 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:37.052449    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0cb53f7-2d7f-423a-ad1c-908410779756" path="/var/lib/kubelet/pods/e0cb53f7-2d7f-423a-ad1c-908410779756/volumes"
Sep 11 18:04:37.054302 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:04:37.054276    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7862-2f4745\" is forbidden: User \"system:node:ip-10-0-138-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-96.ec2.internal' and this object" podUID="e0cb53f7-2d7f-423a-ad1c-908410779756" pod="opendatahub/test-curl-7862-2f4745"
Sep 11 18:04:37.135615 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:37.135568    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqmfv\" (UniqueName: \"kubernetes.io/projected/e4f208ba-4a9f-4aec-ab79-ff7533771e64-kube-api-access-gqmfv\") pod \"test-curl-7862-5d0f8b\" (UID: \"e4f208ba-4a9f-4aec-ab79-ff7533771e64\") " pod="opendatahub/test-curl-7862-5d0f8b"
Sep 11 18:04:37.152134 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:04:37.152105    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7862-9fff8c" container="test-curl-7862-9fff8c"
Sep 11 18:04:37.236656 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:37.236619    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqmfv\" (UniqueName: \"kubernetes.io/projected/e4f208ba-4a9f-4aec-ab79-ff7533771e64-kube-api-access-gqmfv\") pod \"test-curl-7862-5d0f8b\" (UID: \"e4f208ba-4a9f-4aec-ab79-ff7533771e64\") " pod="opendatahub/test-curl-7862-5d0f8b"
Sep 11 18:04:37.246244 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:37.246211    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqmfv\" (UniqueName: \"kubernetes.io/projected/e4f208ba-4a9f-4aec-ab79-ff7533771e64-kube-api-access-gqmfv\") pod \"test-curl-7862-5d0f8b\" (UID: \"e4f208ba-4a9f-4aec-ab79-ff7533771e64\") " pod="opendatahub/test-curl-7862-5d0f8b"
Sep 11 18:04:37.308987 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:37.308959    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7862-a67557"
Sep 11 18:04:37.329877 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:37.329849    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7862-5d0f8b"
Sep 11 18:04:37.439141 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:37.439103    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65hjv\" (UniqueName: \"kubernetes.io/projected/937920d8-8daa-472f-a499-accce6cbbee9-kube-api-access-65hjv\") pod \"937920d8-8daa-472f-a499-accce6cbbee9\" (UID: \"937920d8-8daa-472f-a499-accce6cbbee9\") "
Sep 11 18:04:37.441156 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:37.441132    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937920d8-8daa-472f-a499-accce6cbbee9-kube-api-access-65hjv" (OuterVolumeSpecName: "kube-api-access-65hjv") pod "937920d8-8daa-472f-a499-accce6cbbee9" (UID: "937920d8-8daa-472f-a499-accce6cbbee9"). InnerVolumeSpecName "kube-api-access-65hjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:04:37.540535 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:37.540446    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65hjv\" (UniqueName: \"kubernetes.io/projected/937920d8-8daa-472f-a499-accce6cbbee9-kube-api-access-65hjv\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:04:37.609588 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:37.609557    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7862-5d0f8b"]
Sep 11 18:04:37.612173 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:04:37.612142    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4f208ba_4a9f_4aec_ab79_ff7533771e64.slice/crio-e54406e679afc7408e67b949194f31b9dd9d89a7580f9639a9befcdc92a9692f WatchSource:0}: Error finding container e54406e679afc7408e67b949194f31b9dd9d89a7580f9639a9befcdc92a9692f: Status 404 returned error can't find the container with id e54406e679afc7408e67b949194f31b9dd9d89a7580f9639a9befcdc92a9692f
Sep 11 18:04:38.052455 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:38.052419    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:04:38.158341 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:38.158309    2559 generic.go:356] "Generic (PLEG): container finished" podID="937920d8-8daa-472f-a499-accce6cbbee9" containerID="fd422db58f1b61f489f9ea4417f62be24d3d10124ba940a106c7f5516a6f0dfd" exitCode=137
Sep 11 18:04:38.158785 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:38.158374    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7862-a67557"
Sep 11 18:04:38.158785 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:38.158413    2559 scope.go:117] "RemoveContainer" containerID="fd422db58f1b61f489f9ea4417f62be24d3d10124ba940a106c7f5516a6f0dfd"
Sep 11 18:04:38.160340 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:38.160306    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7862-5d0f8b" event={"ID":"e4f208ba-4a9f-4aec-ab79-ff7533771e64","Type":"ContainerStarted","Data":"c8424c976fd42237493a02de4f4da158e7a8577ac93246d3193c8a127a8f88c9"}
Sep 11 18:04:38.160493 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:38.160412    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7862-5d0f8b" event={"ID":"e4f208ba-4a9f-4aec-ab79-ff7533771e64","Type":"ContainerStarted","Data":"e54406e679afc7408e67b949194f31b9dd9d89a7580f9639a9befcdc92a9692f"}
Sep 11 18:04:38.160547 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:38.160521    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7862-9fff8c" podUID="b59929f6-cb0e-434c-9cc5-948785c84eaa" containerName="test-curl-7862-9fff8c" containerID="cri-o://eeb4936208a306b7bb06e1148313bc962f12e64cbc5c2d4a61b987ca99595a34" gracePeriod=2
Sep 11 18:04:38.168746 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:38.168726    2559 scope.go:117] "RemoveContainer" containerID="fd422db58f1b61f489f9ea4417f62be24d3d10124ba940a106c7f5516a6f0dfd"
Sep 11 18:04:38.169044 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:04:38.169027    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd422db58f1b61f489f9ea4417f62be24d3d10124ba940a106c7f5516a6f0dfd\": container with ID starting with fd422db58f1b61f489f9ea4417f62be24d3d10124ba940a106c7f5516a6f0dfd not found: ID does not exist" containerID="fd422db58f1b61f489f9ea4417f62be24d3d10124ba940a106c7f5516a6f0dfd"
Sep 11 18:04:38.169084 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:38.169057    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd422db58f1b61f489f9ea4417f62be24d3d10124ba940a106c7f5516a6f0dfd"} err="failed to get container status \"fd422db58f1b61f489f9ea4417f62be24d3d10124ba940a106c7f5516a6f0dfd\": rpc error: code = NotFound desc = could not find container \"fd422db58f1b61f489f9ea4417f62be24d3d10124ba940a106c7f5516a6f0dfd\": container with ID starting with fd422db58f1b61f489f9ea4417f62be24d3d10124ba940a106c7f5516a6f0dfd not found: ID does not exist"
Sep 11 18:04:38.182592 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:38.182545    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7862-5d0f8b" podStartSLOduration=2.182530064 podStartE2EDuration="2.182530064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:04:38.181655425 +0000 UTC m=+1183.626755997" watchObservedRunningTime="2026-09-11 18:04:38.182530064 +0000 UTC m=+1183.627630634"
Sep 11 18:04:38.184598 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:04:38.184561    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7862-a67557\" is forbidden: User \"system:node:ip-10-0-138-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-96.ec2.internal' and this object" podUID="937920d8-8daa-472f-a499-accce6cbbee9" pod="opendatahub/test-curl-7862-a67557"
Sep 11 18:04:38.936193 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:38.936157    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7862-5d0f8b"]
Sep 11 18:04:38.945399 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:38.945367    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7862-5d0f8b"]
Sep 11 18:04:39.053384 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:39.053333    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="937920d8-8daa-472f-a499-accce6cbbee9" path="/var/lib/kubelet/pods/937920d8-8daa-472f-a499-accce6cbbee9/volumes"
Sep 11 18:04:39.166568 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:04:39.166537    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7862-5d0f8b" container="test-curl-7862-5d0f8b"
Sep 11 18:04:40.171746 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:40.171715    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7862-5d0f8b" podUID="e4f208ba-4a9f-4aec-ab79-ff7533771e64" containerName="test-curl-7862-5d0f8b" containerID="cri-o://c8424c976fd42237493a02de4f4da158e7a8577ac93246d3193c8a127a8f88c9" gracePeriod=2
Sep 11 18:04:40.350128 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:40.350102    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7862-9fff8c"
Sep 11 18:04:40.362436 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:40.362333    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47ldt\" (UniqueName: \"kubernetes.io/projected/b59929f6-cb0e-434c-9cc5-948785c84eaa-kube-api-access-47ldt\") pod \"b59929f6-cb0e-434c-9cc5-948785c84eaa\" (UID: \"b59929f6-cb0e-434c-9cc5-948785c84eaa\") "
Sep 11 18:04:40.364328 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:40.364295    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b59929f6-cb0e-434c-9cc5-948785c84eaa-kube-api-access-47ldt" (OuterVolumeSpecName: "kube-api-access-47ldt") pod "b59929f6-cb0e-434c-9cc5-948785c84eaa" (UID: "b59929f6-cb0e-434c-9cc5-948785c84eaa"). InnerVolumeSpecName "kube-api-access-47ldt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:04:40.463826 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:40.463739    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47ldt\" (UniqueName: \"kubernetes.io/projected/b59929f6-cb0e-434c-9cc5-948785c84eaa-kube-api-access-47ldt\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:04:41.053410 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:41.053372    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b59929f6-cb0e-434c-9cc5-948785c84eaa" path="/var/lib/kubelet/pods/b59929f6-cb0e-434c-9cc5-948785c84eaa/volumes"
Sep 11 18:04:41.177802 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:41.177760    2559 generic.go:356] "Generic (PLEG): container finished" podID="b59929f6-cb0e-434c-9cc5-948785c84eaa" containerID="eeb4936208a306b7bb06e1148313bc962f12e64cbc5c2d4a61b987ca99595a34" exitCode=137
Sep 11 18:04:41.177802 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:41.177801    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7862-9fff8c"
Sep 11 18:04:41.178288 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:41.177833    2559 scope.go:117] "RemoveContainer" containerID="eeb4936208a306b7bb06e1148313bc962f12e64cbc5c2d4a61b987ca99595a34"
Sep 11 18:04:41.186711 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:41.186691    2559 scope.go:117] "RemoveContainer" containerID="eeb4936208a306b7bb06e1148313bc962f12e64cbc5c2d4a61b987ca99595a34"
Sep 11 18:04:41.186974 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:04:41.186954    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeb4936208a306b7bb06e1148313bc962f12e64cbc5c2d4a61b987ca99595a34\": container with ID starting with eeb4936208a306b7bb06e1148313bc962f12e64cbc5c2d4a61b987ca99595a34 not found: ID does not exist" containerID="eeb4936208a306b7bb06e1148313bc962f12e64cbc5c2d4a61b987ca99595a34"
Sep 11 18:04:41.187021 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:41.186982    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb4936208a306b7bb06e1148313bc962f12e64cbc5c2d4a61b987ca99595a34"} err="failed to get container status \"eeb4936208a306b7bb06e1148313bc962f12e64cbc5c2d4a61b987ca99595a34\": rpc error: code = NotFound desc = could not find container \"eeb4936208a306b7bb06e1148313bc962f12e64cbc5c2d4a61b987ca99595a34\": container with ID starting with eeb4936208a306b7bb06e1148313bc962f12e64cbc5c2d4a61b987ca99595a34 not found: ID does not exist"
Sep 11 18:04:42.362020 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:42.361991    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7862-5d0f8b"
Sep 11 18:04:42.380215 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:42.380193    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqmfv\" (UniqueName: \"kubernetes.io/projected/e4f208ba-4a9f-4aec-ab79-ff7533771e64-kube-api-access-gqmfv\") pod \"e4f208ba-4a9f-4aec-ab79-ff7533771e64\" (UID: \"e4f208ba-4a9f-4aec-ab79-ff7533771e64\") "
Sep 11 18:04:42.382117 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:42.382087    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f208ba-4a9f-4aec-ab79-ff7533771e64-kube-api-access-gqmfv" (OuterVolumeSpecName: "kube-api-access-gqmfv") pod "e4f208ba-4a9f-4aec-ab79-ff7533771e64" (UID: "e4f208ba-4a9f-4aec-ab79-ff7533771e64"). InnerVolumeSpecName "kube-api-access-gqmfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:04:42.481847 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:42.481766    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqmfv\" (UniqueName: \"kubernetes.io/projected/e4f208ba-4a9f-4aec-ab79-ff7533771e64-kube-api-access-gqmfv\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:04:43.053594 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:43.053557    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4f208ba-4a9f-4aec-ab79-ff7533771e64" path="/var/lib/kubelet/pods/e4f208ba-4a9f-4aec-ab79-ff7533771e64/volumes"
Sep 11 18:04:43.191483 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:43.191450    2559 generic.go:356] "Generic (PLEG): container finished" podID="e4f208ba-4a9f-4aec-ab79-ff7533771e64" containerID="c8424c976fd42237493a02de4f4da158e7a8577ac93246d3193c8a127a8f88c9" exitCode=137
Sep 11 18:04:43.191696 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:43.191500    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7862-5d0f8b"
Sep 11 18:04:43.191696 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:43.191553    2559 scope.go:117] "RemoveContainer" containerID="c8424c976fd42237493a02de4f4da158e7a8577ac93246d3193c8a127a8f88c9"
Sep 11 18:04:43.200107 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:43.200090    2559 scope.go:117] "RemoveContainer" containerID="c8424c976fd42237493a02de4f4da158e7a8577ac93246d3193c8a127a8f88c9"
Sep 11 18:04:43.200389 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:04:43.200364    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8424c976fd42237493a02de4f4da158e7a8577ac93246d3193c8a127a8f88c9\": container with ID starting with c8424c976fd42237493a02de4f4da158e7a8577ac93246d3193c8a127a8f88c9 not found: ID does not exist" containerID="c8424c976fd42237493a02de4f4da158e7a8577ac93246d3193c8a127a8f88c9"
Sep 11 18:04:43.200431 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:43.200399    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8424c976fd42237493a02de4f4da158e7a8577ac93246d3193c8a127a8f88c9"} err="failed to get container status \"c8424c976fd42237493a02de4f4da158e7a8577ac93246d3193c8a127a8f88c9\": rpc error: code = NotFound desc = could not find container \"c8424c976fd42237493a02de4f4da158e7a8577ac93246d3193c8a127a8f88c9\": container with ID starting with c8424c976fd42237493a02de4f4da158e7a8577ac93246d3193c8a127a8f88c9 not found: ID does not exist"
Sep 11 18:04:47.922326 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:47.922292    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j"]
Sep 11 18:04:47.922740 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:47.922724    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4f208ba-4a9f-4aec-ab79-ff7533771e64" containerName="test-curl-7862-5d0f8b"
Sep 11 18:04:47.922782 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:47.922741    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4f208ba-4a9f-4aec-ab79-ff7533771e64" containerName="test-curl-7862-5d0f8b"
Sep 11 18:04:47.922834 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:47.922824    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4f208ba-4a9f-4aec-ab79-ff7533771e64" containerName="test-curl-7862-5d0f8b"
Sep 11 18:04:47.927248 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:47.927226    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j"
Sep 11 18:04:47.930935 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:47.930914    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7e61be66-serving-cert\""
Sep 11 18:04:47.944042 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:47.944012    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j"]
Sep 11 18:04:48.028688 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.028642    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnmjk\" (UniqueName: \"kubernetes.io/projected/519955fa-6eef-40a5-ae6c-3fb307acdfb6-kube-api-access-jnmjk\") pod \"maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j\" (UID: \"519955fa-6eef-40a5-ae6c-3fb307acdfb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j"
Sep 11 18:04:48.029058 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.028709    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519955fa-6eef-40a5-ae6c-3fb307acdfb6-maas-api-tls\") pod \"maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j\" (UID: \"519955fa-6eef-40a5-ae6c-3fb307acdfb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j"
Sep 11 18:04:48.029058 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.028753    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519955fa-6eef-40a5-ae6c-3fb307acdfb6-metrics-certs\") pod \"maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j\" (UID: \"519955fa-6eef-40a5-ae6c-3fb307acdfb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j"
Sep 11 18:04:48.029058 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.028826    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519955fa-6eef-40a5-ae6c-3fb307acdfb6-openshift-service-ca\") pod \"maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j\" (UID: \"519955fa-6eef-40a5-ae6c-3fb307acdfb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j"
Sep 11 18:04:48.130285 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.130240    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnmjk\" (UniqueName: \"kubernetes.io/projected/519955fa-6eef-40a5-ae6c-3fb307acdfb6-kube-api-access-jnmjk\") pod \"maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j\" (UID: \"519955fa-6eef-40a5-ae6c-3fb307acdfb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j"
Sep 11 18:04:48.130525 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.130311    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519955fa-6eef-40a5-ae6c-3fb307acdfb6-maas-api-tls\") pod \"maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j\" (UID: \"519955fa-6eef-40a5-ae6c-3fb307acdfb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j"
Sep 11 18:04:48.130525 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.130337    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519955fa-6eef-40a5-ae6c-3fb307acdfb6-metrics-certs\") pod \"maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j\" (UID: \"519955fa-6eef-40a5-ae6c-3fb307acdfb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j"
Sep 11 18:04:48.130525 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.130416    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519955fa-6eef-40a5-ae6c-3fb307acdfb6-openshift-service-ca\") pod \"maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j\" (UID: \"519955fa-6eef-40a5-ae6c-3fb307acdfb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j"
Sep 11 18:04:48.131026 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.131002    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519955fa-6eef-40a5-ae6c-3fb307acdfb6-openshift-service-ca\") pod \"maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j\" (UID: \"519955fa-6eef-40a5-ae6c-3fb307acdfb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j"
Sep 11 18:04:48.132951 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.132917    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519955fa-6eef-40a5-ae6c-3fb307acdfb6-maas-api-tls\") pod \"maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j\" (UID: \"519955fa-6eef-40a5-ae6c-3fb307acdfb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j"
Sep 11 18:04:48.133051 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.132953    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519955fa-6eef-40a5-ae6c-3fb307acdfb6-metrics-certs\") pod \"maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j\" (UID: \"519955fa-6eef-40a5-ae6c-3fb307acdfb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j"
Sep 11 18:04:48.139949 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.139920    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnmjk\" (UniqueName: \"kubernetes.io/projected/519955fa-6eef-40a5-ae6c-3fb307acdfb6-kube-api-access-jnmjk\") pod \"maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j\" (UID: \"519955fa-6eef-40a5-ae6c-3fb307acdfb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j"
Sep 11 18:04:48.238627 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.238526    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j"
Sep 11 18:04:48.525374 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.525326    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j"]
Sep 11 18:04:48.525875 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:04:48.525841    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519955fa_6eef_40a5_ae6c_3fb307acdfb6.slice/crio-f467476d022e9c16e62d5d79c1209b28adac1b23279cfc6012a76bca2314acb7 WatchSource:0}: Error finding container f467476d022e9c16e62d5d79c1209b28adac1b23279cfc6012a76bca2314acb7: Status 404 returned error can't find the container with id f467476d022e9c16e62d5d79c1209b28adac1b23279cfc6012a76bca2314acb7
Sep 11 18:04:48.794741 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.794659    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt"]
Sep 11 18:04:48.799574 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.799552    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt"
Sep 11 18:04:48.802177 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.802151    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7e61be66-dockercfg-75d6h\""
Sep 11 18:04:48.802573 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.802552    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7e61be66\""
Sep 11 18:04:48.824074 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.824038    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt"]
Sep 11 18:04:48.839197 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.839158    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hph74\" (UniqueName: \"kubernetes.io/projected/85a873b6-8f86-4acd-a8e1-b4151a856c43-kube-api-access-hph74\") pod \"payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt\" (UID: \"85a873b6-8f86-4acd-a8e1-b4151a856c43\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt"
Sep 11 18:04:48.839444 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.839240    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a873b6-8f86-4acd-a8e1-b4151a856c43-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt\" (UID: \"85a873b6-8f86-4acd-a8e1-b4151a856c43\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt"
Sep 11 18:04:48.940614 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.940584    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hph74\" (UniqueName: \"kubernetes.io/projected/85a873b6-8f86-4acd-a8e1-b4151a856c43-kube-api-access-hph74\") pod \"payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt\" (UID: \"85a873b6-8f86-4acd-a8e1-b4151a856c43\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt"
Sep 11 18:04:48.941112 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.940682    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a873b6-8f86-4acd-a8e1-b4151a856c43-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt\" (UID: \"85a873b6-8f86-4acd-a8e1-b4151a856c43\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt"
Sep 11 18:04:48.941472 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.941443    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a873b6-8f86-4acd-a8e1-b4151a856c43-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt\" (UID: \"85a873b6-8f86-4acd-a8e1-b4151a856c43\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt"
Sep 11 18:04:48.961265 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:48.961232    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hph74\" (UniqueName: \"kubernetes.io/projected/85a873b6-8f86-4acd-a8e1-b4151a856c43-kube-api-access-hph74\") pod \"payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt\" (UID: \"85a873b6-8f86-4acd-a8e1-b4151a856c43\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt"
Sep 11 18:04:49.109804 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:49.109712    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt"
Sep 11 18:04:49.228396 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:49.228322    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j" event={"ID":"519955fa-6eef-40a5-ae6c-3fb307acdfb6","Type":"ContainerStarted","Data":"6bd93a8bc3feab1086327ca035f35976fd8053f74b300256d9d69dbc6ca17b43"}
Sep 11 18:04:49.228599 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:49.228426    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j" event={"ID":"519955fa-6eef-40a5-ae6c-3fb307acdfb6","Type":"ContainerStarted","Data":"f467476d022e9c16e62d5d79c1209b28adac1b23279cfc6012a76bca2314acb7"}
Sep 11 18:04:49.228599 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:49.228542    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j"
Sep 11 18:04:49.392098 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:49.392045    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j" podStartSLOduration=2.0293006 podStartE2EDuration="2.392025062s" totalImagesPullingTime="362.724462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:04:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:04:49.263493633 +0000 UTC m=+1194.708594204" watchObservedRunningTime="2026-09-11 18:04:49.392025062 +0000 UTC m=+1194.837125655"
Sep 11 18:04:49.394432 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:49.394407    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt"]
Sep 11 18:04:49.396159 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:04:49.396127    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a873b6_8f86_4acd_a8e1_b4151a856c43.slice/crio-3fc1288257fa92ca70b49fbcf47edbdf20ac3a1c78bada3834dc00d5e92feefb WatchSource:0}: Error finding container 3fc1288257fa92ca70b49fbcf47edbdf20ac3a1c78bada3834dc00d5e92feefb: Status 404 returned error can't find the container with id 3fc1288257fa92ca70b49fbcf47edbdf20ac3a1c78bada3834dc00d5e92feefb
Sep 11 18:04:50.236623 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:50.236589    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt" event={"ID":"85a873b6-8f86-4acd-a8e1-b4151a856c43","Type":"ContainerStarted","Data":"3fc1288257fa92ca70b49fbcf47edbdf20ac3a1c78bada3834dc00d5e92feefb"}
Sep 11 18:04:51.244140 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:51.244102    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt" event={"ID":"85a873b6-8f86-4acd-a8e1-b4151a856c43","Type":"ContainerStarted","Data":"e0081504d45687c08533390f3bfe65656c31b8967c0b13a6b97ee2dea2bf0b08"}
Sep 11 18:04:51.244620 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:51.244193    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt"
Sep 11 18:04:51.271217 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:51.271154    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt" podStartSLOduration=1.609808313 podStartE2EDuration="3.271134981s" totalImagesPullingTime="1.661326668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:04:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:04:51.268835896 +0000 UTC m=+1196.713936493" watchObservedRunningTime="2026-09-11 18:04:51.271134981 +0000 UTC m=+1196.716235588"
Sep 11 18:04:52.664433 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:52.664393    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:04:55.242025 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:55.241995    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j"
Sep 11 18:04:57.829335 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:57.829285    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"]
Sep 11 18:04:57.835790 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:57.835756    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:04:57.841716 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:57.841679    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-386935\"/\"openshift-service-ca.crt\""
Sep 11 18:04:57.842491 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:57.842460    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-386935\"/\"auth-test-model-386935-kserve-self-signed-certs\""
Sep 11 18:04:57.848669 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:57.848641    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-386935\"/\"default-dockercfg-2787z\""
Sep 11 18:04:57.849236 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:57.849192    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-386935\"/\"kube-root-ca.crt\""
Sep 11 18:04:57.872915 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:57.872877    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"]
Sep 11 18:04:57.930008 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:57.929957    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-model-cache\") pod \"auth-test-model-386935-kserve-69dd649579-bbjkf\" (UID: \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\") " pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:04:57.930211 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:57.930028    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-tmp-dir\") pod \"auth-test-model-386935-kserve-69dd649579-bbjkf\" (UID: \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\") " pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:04:57.930211 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:57.930066    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-home\") pod \"auth-test-model-386935-kserve-69dd649579-bbjkf\" (UID: \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\") " pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:04:57.930211 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:57.930092    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pkgc\" (UniqueName: \"kubernetes.io/projected/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-kube-api-access-4pkgc\") pod \"auth-test-model-386935-kserve-69dd649579-bbjkf\" (UID: \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\") " pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:04:57.930211 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:57.930153    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-dshm\") pod \"auth-test-model-386935-kserve-69dd649579-bbjkf\" (UID: \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\") " pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:04:57.930401 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:57.930214    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-tls-certs\") pod \"auth-test-model-386935-kserve-69dd649579-bbjkf\" (UID: \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\") " pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:04:58.031502 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:58.031460    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-model-cache\") pod \"auth-test-model-386935-kserve-69dd649579-bbjkf\" (UID: \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\") " pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:04:58.031692 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:58.031523    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-tmp-dir\") pod \"auth-test-model-386935-kserve-69dd649579-bbjkf\" (UID: \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\") " pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:04:58.031692 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:58.031555    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-home\") pod \"auth-test-model-386935-kserve-69dd649579-bbjkf\" (UID: \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\") " pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:04:58.031692 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:58.031572    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pkgc\" (UniqueName: \"kubernetes.io/projected/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-kube-api-access-4pkgc\") pod \"auth-test-model-386935-kserve-69dd649579-bbjkf\" (UID: \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\") " pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:04:58.031692 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:58.031595    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-dshm\") pod \"auth-test-model-386935-kserve-69dd649579-bbjkf\" (UID: \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\") " pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:04:58.031692 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:58.031642    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-tls-certs\") pod \"auth-test-model-386935-kserve-69dd649579-bbjkf\" (UID: \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\") " pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:04:58.031970 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:58.031943    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-model-cache\") pod \"auth-test-model-386935-kserve-69dd649579-bbjkf\" (UID: \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\") " pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:04:58.032022 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:58.031991    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-home\") pod \"auth-test-model-386935-kserve-69dd649579-bbjkf\" (UID: \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\") " pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:04:58.032056 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:58.032001    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-tmp-dir\") pod \"auth-test-model-386935-kserve-69dd649579-bbjkf\" (UID: \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\") " pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:04:58.034010 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:58.033982    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-dshm\") pod \"auth-test-model-386935-kserve-69dd649579-bbjkf\" (UID: \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\") " pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:04:58.034263 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:58.034244    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-tls-certs\") pod \"auth-test-model-386935-kserve-69dd649579-bbjkf\" (UID: \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\") " pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:04:58.046857 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:58.046826    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pkgc\" (UniqueName: \"kubernetes.io/projected/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-kube-api-access-4pkgc\") pod \"auth-test-model-386935-kserve-69dd649579-bbjkf\" (UID: \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\") " pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:04:58.145291 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:58.145251    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:04:58.424812 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:58.424784    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"]
Sep 11 18:04:58.427025 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:04:58.426989    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7174b0e2_83f4_4ebf_bb60_6f7f3a93015b.slice/crio-76411924a71bd743df337cf87965501ddd958a119686eb2a53a0c74cd1f0a2dd WatchSource:0}: Error finding container 76411924a71bd743df337cf87965501ddd958a119686eb2a53a0c74cd1f0a2dd: Status 404 returned error can't find the container with id 76411924a71bd743df337cf87965501ddd958a119686eb2a53a0c74cd1f0a2dd
Sep 11 18:04:59.292846 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:59.292806    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf" event={"ID":"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b","Type":"ContainerStarted","Data":"9c1e11dfeb41e6bdd85d995c0f06aa91f05aaae9afaaea4452cb4dd2fd75e8b5"}
Sep 11 18:04:59.292846 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:59.292853    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf" event={"ID":"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b","Type":"ContainerStarted","Data":"76411924a71bd743df337cf87965501ddd958a119686eb2a53a0c74cd1f0a2dd"}
Sep 11 18:04:59.319124 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:04:59.319068    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf" podStartSLOduration=2.319053785 podStartE2EDuration="2.319053785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:04:59.31868417 +0000 UTC m=+1204.763784741" watchObservedRunningTime="2026-09-11 18:04:59.319053785 +0000 UTC m=+1204.764154355"
Sep 11 18:05:02.252735 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:02.252705    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt"
Sep 11 18:05:08.145656 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:08.145610    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:05:08.146109 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:08.145694    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:05:08.158445 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:08.158409    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:05:08.358742 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:08.358712    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:05:10.854496 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:10.854455    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:05:29.870285 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:29.870244    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"]
Sep 11 18:05:29.874395 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:29.874371    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:05:29.877668 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:29.877638    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3ebea5\"/\"kube-root-ca.crt\""
Sep 11 18:05:29.878391 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:29.878335    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3ebea5\"/\"default-dockercfg-5wbch\""
Sep 11 18:05:29.878585 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:29.878565    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3ebea5\"/\"auth-test-model-3ebea5-kserve-self-signed-certs\""
Sep 11 18:05:29.878668 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:29.878648    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3ebea5\"/\"openshift-service-ca.crt\""
Sep 11 18:05:29.892169 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:29.892137    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"]
Sep 11 18:05:29.950482 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:29.950448    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81016762-cd02-4fda-a227-5b191dfd4d1d-tls-certs\") pod \"auth-test-model-3ebea5-kserve-f5788cddc-mg96w\" (UID: \"81016762-cd02-4fda-a227-5b191dfd4d1d\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:05:29.950654 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:29.950497    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81016762-cd02-4fda-a227-5b191dfd4d1d-dshm\") pod \"auth-test-model-3ebea5-kserve-f5788cddc-mg96w\" (UID: \"81016762-cd02-4fda-a227-5b191dfd4d1d\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:05:29.950654 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:29.950570    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81016762-cd02-4fda-a227-5b191dfd4d1d-model-cache\") pod \"auth-test-model-3ebea5-kserve-f5788cddc-mg96w\" (UID: \"81016762-cd02-4fda-a227-5b191dfd4d1d\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:05:29.950654 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:29.950630    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7nzs\" (UniqueName: \"kubernetes.io/projected/81016762-cd02-4fda-a227-5b191dfd4d1d-kube-api-access-s7nzs\") pod \"auth-test-model-3ebea5-kserve-f5788cddc-mg96w\" (UID: \"81016762-cd02-4fda-a227-5b191dfd4d1d\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:05:29.950774 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:29.950696    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81016762-cd02-4fda-a227-5b191dfd4d1d-tmp-dir\") pod \"auth-test-model-3ebea5-kserve-f5788cddc-mg96w\" (UID: \"81016762-cd02-4fda-a227-5b191dfd4d1d\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:05:29.950774 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:29.950737    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81016762-cd02-4fda-a227-5b191dfd4d1d-home\") pod \"auth-test-model-3ebea5-kserve-f5788cddc-mg96w\" (UID: \"81016762-cd02-4fda-a227-5b191dfd4d1d\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:05:30.051857 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:30.051821    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81016762-cd02-4fda-a227-5b191dfd4d1d-tls-certs\") pod \"auth-test-model-3ebea5-kserve-f5788cddc-mg96w\" (UID: \"81016762-cd02-4fda-a227-5b191dfd4d1d\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:05:30.051857 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:30.051869    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81016762-cd02-4fda-a227-5b191dfd4d1d-dshm\") pod \"auth-test-model-3ebea5-kserve-f5788cddc-mg96w\" (UID: \"81016762-cd02-4fda-a227-5b191dfd4d1d\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:05:30.052124 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:30.051978    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81016762-cd02-4fda-a227-5b191dfd4d1d-model-cache\") pod \"auth-test-model-3ebea5-kserve-f5788cddc-mg96w\" (UID: \"81016762-cd02-4fda-a227-5b191dfd4d1d\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:05:30.052124 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:30.052031    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7nzs\" (UniqueName: \"kubernetes.io/projected/81016762-cd02-4fda-a227-5b191dfd4d1d-kube-api-access-s7nzs\") pod \"auth-test-model-3ebea5-kserve-f5788cddc-mg96w\" (UID: \"81016762-cd02-4fda-a227-5b191dfd4d1d\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:05:30.052124 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:30.052102    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81016762-cd02-4fda-a227-5b191dfd4d1d-tmp-dir\") pod \"auth-test-model-3ebea5-kserve-f5788cddc-mg96w\" (UID: \"81016762-cd02-4fda-a227-5b191dfd4d1d\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:05:30.052297 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:30.052153    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81016762-cd02-4fda-a227-5b191dfd4d1d-home\") pod \"auth-test-model-3ebea5-kserve-f5788cddc-mg96w\" (UID: \"81016762-cd02-4fda-a227-5b191dfd4d1d\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:05:30.052444 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:30.052425    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81016762-cd02-4fda-a227-5b191dfd4d1d-model-cache\") pod \"auth-test-model-3ebea5-kserve-f5788cddc-mg96w\" (UID: \"81016762-cd02-4fda-a227-5b191dfd4d1d\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:05:30.052537 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:30.052515    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81016762-cd02-4fda-a227-5b191dfd4d1d-tmp-dir\") pod \"auth-test-model-3ebea5-kserve-f5788cddc-mg96w\" (UID: \"81016762-cd02-4fda-a227-5b191dfd4d1d\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:05:30.052584 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:30.052551    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81016762-cd02-4fda-a227-5b191dfd4d1d-home\") pod \"auth-test-model-3ebea5-kserve-f5788cddc-mg96w\" (UID: \"81016762-cd02-4fda-a227-5b191dfd4d1d\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:05:30.054224 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:30.054206    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81016762-cd02-4fda-a227-5b191dfd4d1d-dshm\") pod \"auth-test-model-3ebea5-kserve-f5788cddc-mg96w\" (UID: \"81016762-cd02-4fda-a227-5b191dfd4d1d\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:05:30.054406 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:30.054390    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81016762-cd02-4fda-a227-5b191dfd4d1d-tls-certs\") pod \"auth-test-model-3ebea5-kserve-f5788cddc-mg96w\" (UID: \"81016762-cd02-4fda-a227-5b191dfd4d1d\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:05:30.061369 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:30.061331    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7nzs\" (UniqueName: \"kubernetes.io/projected/81016762-cd02-4fda-a227-5b191dfd4d1d-kube-api-access-s7nzs\") pod \"auth-test-model-3ebea5-kserve-f5788cddc-mg96w\" (UID: \"81016762-cd02-4fda-a227-5b191dfd4d1d\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:05:30.184623 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:30.184587    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:05:30.460598 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:30.460568    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"]
Sep 11 18:05:30.462304 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:05:30.462276    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81016762_cd02_4fda_a227_5b191dfd4d1d.slice/crio-606b07e17d3fa4efc90116f433022f4e89df21759503d6f6b77ff4eb68862328 WatchSource:0}: Error finding container 606b07e17d3fa4efc90116f433022f4e89df21759503d6f6b77ff4eb68862328: Status 404 returned error can't find the container with id 606b07e17d3fa4efc90116f433022f4e89df21759503d6f6b77ff4eb68862328
Sep 11 18:05:30.469507 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:30.469470    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w" event={"ID":"81016762-cd02-4fda-a227-5b191dfd4d1d","Type":"ContainerStarted","Data":"606b07e17d3fa4efc90116f433022f4e89df21759503d6f6b77ff4eb68862328"}
Sep 11 18:05:31.476998 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:31.476957    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w" event={"ID":"81016762-cd02-4fda-a227-5b191dfd4d1d","Type":"ContainerStarted","Data":"b9907f9bacff9fc2df9bd08916de917c26f30942d6df95852a1b108dee87bc88"}
Sep 11 18:05:31.510074 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:31.510021    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w" podStartSLOduration=2.510003614 podStartE2EDuration="2.510003614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:05:31.509660662 +0000 UTC m=+1236.954761233" watchObservedRunningTime="2026-09-11 18:05:31.510003614 +0000 UTC m=+1236.955104184"
Sep 11 18:05:34.680322 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:34.680273    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:05:40.185385 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:40.185329    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:05:40.185808 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:40.185400    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:05:40.198152 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:40.198118    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:05:40.537112 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:40.537027    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:05:44.077301 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.077259    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"]
Sep 11 18:05:44.081345 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.081322    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:05:44.086188 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.086164    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7e61be66\"/\"kube-root-ca.crt\""
Sep 11 18:05:44.087325 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.087303    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7e61be66\"/\"e2e-lifecycle-model-7e61be66-kserve-self-signed-certs\""
Sep 11 18:05:44.087464 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.087377    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7e61be66\"/\"openshift-service-ca.crt\""
Sep 11 18:05:44.087464 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.087386    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7e61be66\"/\"default-dockercfg-mwchk\""
Sep 11 18:05:44.101695 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.101666    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"]
Sep 11 18:05:44.194152 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.194113    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-tls-certs\") pod \"e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh\" (UID: \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\") " pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:05:44.194379 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.194232    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5szcz\" (UniqueName: \"kubernetes.io/projected/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-kube-api-access-5szcz\") pod \"e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh\" (UID: \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\") " pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:05:44.194379 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.194289    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-dshm\") pod \"e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh\" (UID: \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\") " pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:05:44.194379 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.194320    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-model-cache\") pod \"e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh\" (UID: \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\") " pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:05:44.194525 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.194466    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-home\") pod \"e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh\" (UID: \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\") " pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:05:44.194568 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.194556    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-tmp-dir\") pod \"e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh\" (UID: \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\") " pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:05:44.295485 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.295445    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5szcz\" (UniqueName: \"kubernetes.io/projected/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-kube-api-access-5szcz\") pod \"e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh\" (UID: \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\") " pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:05:44.295705 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.295502    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-dshm\") pod \"e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh\" (UID: \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\") " pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:05:44.295705 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.295527    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-model-cache\") pod \"e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh\" (UID: \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\") " pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:05:44.295705 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.295604    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-home\") pod \"e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh\" (UID: \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\") " pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:05:44.295705 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.295668    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-tmp-dir\") pod \"e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh\" (UID: \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\") " pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:05:44.295910 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.295813    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-tls-certs\") pod \"e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh\" (UID: \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\") " pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:05:44.296083 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.296053    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-model-cache\") pod \"e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh\" (UID: \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\") " pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:05:44.296227 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.296061    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-home\") pod \"e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh\" (UID: \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\") " pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:05:44.296227 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.296139    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-tmp-dir\") pod \"e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh\" (UID: \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\") " pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:05:44.297946 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.297923    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-dshm\") pod \"e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh\" (UID: \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\") " pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:05:44.298521 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.298498    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-tls-certs\") pod \"e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh\" (UID: \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\") " pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:05:44.305565 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.305536    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5szcz\" (UniqueName: \"kubernetes.io/projected/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-kube-api-access-5szcz\") pod \"e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh\" (UID: \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\") " pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:05:44.391581 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.391547    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:05:44.681574 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:44.681547    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"]
Sep 11 18:05:44.682174 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:05:44.682150    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa59b260_9f5e_40a9_a6b6_37d54b3e550a.slice/crio-63380bb5308ae53824210e8368ee6bfcac547518c57ebdf8b7f97ba3c6898e26 WatchSource:0}: Error finding container 63380bb5308ae53824210e8368ee6bfcac547518c57ebdf8b7f97ba3c6898e26: Status 404 returned error can't find the container with id 63380bb5308ae53824210e8368ee6bfcac547518c57ebdf8b7f97ba3c6898e26
Sep 11 18:05:45.556141 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:45.556101    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh" event={"ID":"aa59b260-9f5e-40a9-a6b6-37d54b3e550a","Type":"ContainerStarted","Data":"c8f59d828eeb5a8c404015a365ddc7cdc9198215f9c1d95d2820a8af4f43cace"}
Sep 11 18:05:45.556682 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:45.556150    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh" event={"ID":"aa59b260-9f5e-40a9-a6b6-37d54b3e550a","Type":"ContainerStarted","Data":"63380bb5308ae53824210e8368ee6bfcac547518c57ebdf8b7f97ba3c6898e26"}
Sep 11 18:05:45.587241 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:45.587185    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh" podStartSLOduration=1.587169787 podStartE2EDuration="1.587169787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:05:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:05:45.585967633 +0000 UTC m=+1251.031068204" watchObservedRunningTime="2026-09-11 18:05:45.587169787 +0000 UTC m=+1251.032270358"
Sep 11 18:05:52.453173 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:52.453077    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:05:54.391780 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:54.391738    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:05:54.392195 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:54.391796    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:05:54.404292 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:54.404259    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:05:54.616974 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:05:54.616941    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:06:10.260137 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:10.260103    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:06:28.702605 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:28.702569    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:06:39.678645 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:39.678613    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j"]
Sep 11 18:06:39.679179 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:39.678820    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j" podUID="519955fa-6eef-40a5-ae6c-3fb307acdfb6" containerName="maas-api" containerID="cri-o://6bd93a8bc3feab1086327ca035f35976fd8053f74b300256d9d69dbc6ca17b43" gracePeriod=30
Sep 11 18:06:39.813372 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:39.813312    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt"]
Sep 11 18:06:39.815050 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:39.813874    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt" podUID="85a873b6-8f86-4acd-a8e1-b4151a856c43" containerName="payload-pre-processing" containerID="cri-o://e0081504d45687c08533390f3bfe65656c31b8967c0b13a6b97ee2dea2bf0b08" gracePeriod=30
Sep 11 18:06:39.872766 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:39.872727    2559 generic.go:356] "Generic (PLEG): container finished" podID="519955fa-6eef-40a5-ae6c-3fb307acdfb6" containerID="6bd93a8bc3feab1086327ca035f35976fd8053f74b300256d9d69dbc6ca17b43" exitCode=0
Sep 11 18:06:39.872939 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:39.872806    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j" event={"ID":"519955fa-6eef-40a5-ae6c-3fb307acdfb6","Type":"ContainerDied","Data":"6bd93a8bc3feab1086327ca035f35976fd8053f74b300256d9d69dbc6ca17b43"}
Sep 11 18:06:39.943163 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:39.943096    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j"
Sep 11 18:06:40.031898 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.031864    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnmjk\" (UniqueName: \"kubernetes.io/projected/519955fa-6eef-40a5-ae6c-3fb307acdfb6-kube-api-access-jnmjk\") pod \"519955fa-6eef-40a5-ae6c-3fb307acdfb6\" (UID: \"519955fa-6eef-40a5-ae6c-3fb307acdfb6\") "
Sep 11 18:06:40.032095 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.031965    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519955fa-6eef-40a5-ae6c-3fb307acdfb6-openshift-service-ca\") pod \"519955fa-6eef-40a5-ae6c-3fb307acdfb6\" (UID: \"519955fa-6eef-40a5-ae6c-3fb307acdfb6\") "
Sep 11 18:06:40.032095 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.032001    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519955fa-6eef-40a5-ae6c-3fb307acdfb6-metrics-certs\") pod \"519955fa-6eef-40a5-ae6c-3fb307acdfb6\" (UID: \"519955fa-6eef-40a5-ae6c-3fb307acdfb6\") "
Sep 11 18:06:40.032095 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.032050    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519955fa-6eef-40a5-ae6c-3fb307acdfb6-maas-api-tls\") pod \"519955fa-6eef-40a5-ae6c-3fb307acdfb6\" (UID: \"519955fa-6eef-40a5-ae6c-3fb307acdfb6\") "
Sep 11 18:06:40.032424 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.032396    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519955fa-6eef-40a5-ae6c-3fb307acdfb6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "519955fa-6eef-40a5-ae6c-3fb307acdfb6" (UID: "519955fa-6eef-40a5-ae6c-3fb307acdfb6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:06:40.035975 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.035935    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519955fa-6eef-40a5-ae6c-3fb307acdfb6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "519955fa-6eef-40a5-ae6c-3fb307acdfb6" (UID: "519955fa-6eef-40a5-ae6c-3fb307acdfb6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:06:40.036116 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.035995    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519955fa-6eef-40a5-ae6c-3fb307acdfb6-kube-api-access-jnmjk" (OuterVolumeSpecName: "kube-api-access-jnmjk") pod "519955fa-6eef-40a5-ae6c-3fb307acdfb6" (UID: "519955fa-6eef-40a5-ae6c-3fb307acdfb6"). InnerVolumeSpecName "kube-api-access-jnmjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:06:40.036116 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.036037    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519955fa-6eef-40a5-ae6c-3fb307acdfb6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "519955fa-6eef-40a5-ae6c-3fb307acdfb6" (UID: "519955fa-6eef-40a5-ae6c-3fb307acdfb6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:06:40.057400 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.057376    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt"
Sep 11 18:06:40.133182 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.133146    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hph74\" (UniqueName: \"kubernetes.io/projected/85a873b6-8f86-4acd-a8e1-b4151a856c43-kube-api-access-hph74\") pod \"85a873b6-8f86-4acd-a8e1-b4151a856c43\" (UID: \"85a873b6-8f86-4acd-a8e1-b4151a856c43\") "
Sep 11 18:06:40.133182 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.133186    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a873b6-8f86-4acd-a8e1-b4151a856c43-plugins-config-volume\") pod \"85a873b6-8f86-4acd-a8e1-b4151a856c43\" (UID: \"85a873b6-8f86-4acd-a8e1-b4151a856c43\") "
Sep 11 18:06:40.133420 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.133367    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519955fa-6eef-40a5-ae6c-3fb307acdfb6-openshift-service-ca\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:06:40.133420 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.133378    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519955fa-6eef-40a5-ae6c-3fb307acdfb6-metrics-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:06:40.133420 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.133387    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519955fa-6eef-40a5-ae6c-3fb307acdfb6-maas-api-tls\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:06:40.133420 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.133395    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnmjk\" (UniqueName: \"kubernetes.io/projected/519955fa-6eef-40a5-ae6c-3fb307acdfb6-kube-api-access-jnmjk\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:06:40.133627 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.133602    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a873b6-8f86-4acd-a8e1-b4151a856c43-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "85a873b6-8f86-4acd-a8e1-b4151a856c43" (UID: "85a873b6-8f86-4acd-a8e1-b4151a856c43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:06:40.135357 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.135327    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a873b6-8f86-4acd-a8e1-b4151a856c43-kube-api-access-hph74" (OuterVolumeSpecName: "kube-api-access-hph74") pod "85a873b6-8f86-4acd-a8e1-b4151a856c43" (UID: "85a873b6-8f86-4acd-a8e1-b4151a856c43"). InnerVolumeSpecName "kube-api-access-hph74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:06:40.234594 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.234504    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hph74\" (UniqueName: \"kubernetes.io/projected/85a873b6-8f86-4acd-a8e1-b4151a856c43-kube-api-access-hph74\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:06:40.234594 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.234536    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a873b6-8f86-4acd-a8e1-b4151a856c43-plugins-config-volume\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:06:40.882888 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.882851    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j"
Sep 11 18:06:40.882888 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.882876    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j" event={"ID":"519955fa-6eef-40a5-ae6c-3fb307acdfb6","Type":"ContainerDied","Data":"f467476d022e9c16e62d5d79c1209b28adac1b23279cfc6012a76bca2314acb7"}
Sep 11 18:06:40.883442 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.882948    2559 scope.go:117] "RemoveContainer" containerID="6bd93a8bc3feab1086327ca035f35976fd8053f74b300256d9d69dbc6ca17b43"
Sep 11 18:06:40.884506 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.884481    2559 generic.go:356] "Generic (PLEG): container finished" podID="85a873b6-8f86-4acd-a8e1-b4151a856c43" containerID="e0081504d45687c08533390f3bfe65656c31b8967c0b13a6b97ee2dea2bf0b08" exitCode=0
Sep 11 18:06:40.884575 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.884533    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt"
Sep 11 18:06:40.884575 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.884565    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt" event={"ID":"85a873b6-8f86-4acd-a8e1-b4151a856c43","Type":"ContainerDied","Data":"e0081504d45687c08533390f3bfe65656c31b8967c0b13a6b97ee2dea2bf0b08"}
Sep 11 18:06:40.884651 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.884606    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt" event={"ID":"85a873b6-8f86-4acd-a8e1-b4151a856c43","Type":"ContainerDied","Data":"3fc1288257fa92ca70b49fbcf47edbdf20ac3a1c78bada3834dc00d5e92feefb"}
Sep 11 18:06:40.893480 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.893463    2559 scope.go:117] "RemoveContainer" containerID="e0081504d45687c08533390f3bfe65656c31b8967c0b13a6b97ee2dea2bf0b08"
Sep 11 18:06:40.902055 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.902039    2559 scope.go:117] "RemoveContainer" containerID="e0081504d45687c08533390f3bfe65656c31b8967c0b13a6b97ee2dea2bf0b08"
Sep 11 18:06:40.902340 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:06:40.902319    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0081504d45687c08533390f3bfe65656c31b8967c0b13a6b97ee2dea2bf0b08\": container with ID starting with e0081504d45687c08533390f3bfe65656c31b8967c0b13a6b97ee2dea2bf0b08 not found: ID does not exist" containerID="e0081504d45687c08533390f3bfe65656c31b8967c0b13a6b97ee2dea2bf0b08"
Sep 11 18:06:40.902457 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.902364    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0081504d45687c08533390f3bfe65656c31b8967c0b13a6b97ee2dea2bf0b08"} err="failed to get container status \"e0081504d45687c08533390f3bfe65656c31b8967c0b13a6b97ee2dea2bf0b08\": rpc error: code = NotFound desc = could not find container \"e0081504d45687c08533390f3bfe65656c31b8967c0b13a6b97ee2dea2bf0b08\": container with ID starting with e0081504d45687c08533390f3bfe65656c31b8967c0b13a6b97ee2dea2bf0b08 not found: ID does not exist"
Sep 11 18:06:40.951212 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.951116    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt"]
Sep 11 18:06:40.955543 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.955513    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7e61be66-7c6b49c894-rcxdt"]
Sep 11 18:06:40.982223 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.982191    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j"]
Sep 11 18:06:40.984469 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:40.984446    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7e61be66-5f879d9db6-gw82j"]
Sep 11 18:06:41.053683 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:41.053652    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="519955fa-6eef-40a5-ae6c-3fb307acdfb6" path="/var/lib/kubelet/pods/519955fa-6eef-40a5-ae6c-3fb307acdfb6/volumes"
Sep 11 18:06:41.054028 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:41.054015    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85a873b6-8f86-4acd-a8e1-b4151a856c43" path="/var/lib/kubelet/pods/85a873b6-8f86-4acd-a8e1-b4151a856c43/volumes"
Sep 11 18:06:46.492473 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:46.492440    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:06:48.887178 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:06:48.887142    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"]
Sep 11 18:07:02.093497 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:02.093460    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:07:03.906211 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:03.906165    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh" podUID="aa59b260-9f5e-40a9-a6b6-37d54b3e550a" containerName="main" containerID="cri-o://c8f59d828eeb5a8c404015a365ddc7cdc9198215f9c1d95d2820a8af4f43cace" gracePeriod=45
Sep 11 18:07:04.156519 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:04.156460    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:07:04.276115 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:04.276085    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5szcz\" (UniqueName: \"kubernetes.io/projected/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-kube-api-access-5szcz\") pod \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\" (UID: \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\") "
Sep 11 18:07:04.276295 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:04.276132    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-home\") pod \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\" (UID: \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\") "
Sep 11 18:07:04.276295 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:04.276151    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-dshm\") pod \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\" (UID: \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\") "
Sep 11 18:07:04.276295 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:04.276172    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-tmp-dir\") pod \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\" (UID: \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\") "
Sep 11 18:07:04.276295 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:04.276209    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-tls-certs\") pod \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\" (UID: \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\") "
Sep 11 18:07:04.276295 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:04.276237    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-model-cache\") pod \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\" (UID: \"aa59b260-9f5e-40a9-a6b6-37d54b3e550a\") "
Sep 11 18:07:04.276601 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:04.276413    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-home" (OuterVolumeSpecName: "home") pod "aa59b260-9f5e-40a9-a6b6-37d54b3e550a" (UID: "aa59b260-9f5e-40a9-a6b6-37d54b3e550a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:07:04.276601 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:04.276527    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa59b260-9f5e-40a9-a6b6-37d54b3e550a" (UID: "aa59b260-9f5e-40a9-a6b6-37d54b3e550a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:07:04.276601 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:04.276590    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-home\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:07:04.276748 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:04.276649    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa59b260-9f5e-40a9-a6b6-37d54b3e550a" (UID: "aa59b260-9f5e-40a9-a6b6-37d54b3e550a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:07:04.278410 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:04.278386    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-dshm" (OuterVolumeSpecName: "dshm") pod "aa59b260-9f5e-40a9-a6b6-37d54b3e550a" (UID: "aa59b260-9f5e-40a9-a6b6-37d54b3e550a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:07:04.278528 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:04.278402    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-kube-api-access-5szcz" (OuterVolumeSpecName: "kube-api-access-5szcz") pod "aa59b260-9f5e-40a9-a6b6-37d54b3e550a" (UID: "aa59b260-9f5e-40a9-a6b6-37d54b3e550a"). InnerVolumeSpecName "kube-api-access-5szcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:07:04.278528 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:04.278469    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa59b260-9f5e-40a9-a6b6-37d54b3e550a" (UID: "aa59b260-9f5e-40a9-a6b6-37d54b3e550a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:07:04.377696 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:04.377659    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-tls-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:07:04.377696 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:04.377689    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-model-cache\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:07:04.377696 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:04.377698    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5szcz\" (UniqueName: \"kubernetes.io/projected/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-kube-api-access-5szcz\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:07:04.377927 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:04.377708    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-dshm\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:07:04.377927 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:04.377716    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa59b260-9f5e-40a9-a6b6-37d54b3e550a-tmp-dir\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:07:05.021484 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:05.021454    2559 generic.go:356] "Generic (PLEG): container finished" podID="aa59b260-9f5e-40a9-a6b6-37d54b3e550a" containerID="c8f59d828eeb5a8c404015a365ddc7cdc9198215f9c1d95d2820a8af4f43cace" exitCode=0
Sep 11 18:07:05.021943 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:05.021487    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh" event={"ID":"aa59b260-9f5e-40a9-a6b6-37d54b3e550a","Type":"ContainerDied","Data":"c8f59d828eeb5a8c404015a365ddc7cdc9198215f9c1d95d2820a8af4f43cace"}
Sep 11 18:07:05.021943 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:05.021513    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"
Sep 11 18:07:05.021943 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:05.021536    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh" event={"ID":"aa59b260-9f5e-40a9-a6b6-37d54b3e550a","Type":"ContainerDied","Data":"63380bb5308ae53824210e8368ee6bfcac547518c57ebdf8b7f97ba3c6898e26"}
Sep 11 18:07:05.021943 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:05.021555    2559 scope.go:117] "RemoveContainer" containerID="c8f59d828eeb5a8c404015a365ddc7cdc9198215f9c1d95d2820a8af4f43cace"
Sep 11 18:07:05.031274 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:05.031249    2559 scope.go:117] "RemoveContainer" containerID="c8f59d828eeb5a8c404015a365ddc7cdc9198215f9c1d95d2820a8af4f43cace"
Sep 11 18:07:05.031572 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:07:05.031553    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f59d828eeb5a8c404015a365ddc7cdc9198215f9c1d95d2820a8af4f43cace\": container with ID starting with c8f59d828eeb5a8c404015a365ddc7cdc9198215f9c1d95d2820a8af4f43cace not found: ID does not exist" containerID="c8f59d828eeb5a8c404015a365ddc7cdc9198215f9c1d95d2820a8af4f43cace"
Sep 11 18:07:05.031647 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:05.031587    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f59d828eeb5a8c404015a365ddc7cdc9198215f9c1d95d2820a8af4f43cace"} err="failed to get container status \"c8f59d828eeb5a8c404015a365ddc7cdc9198215f9c1d95d2820a8af4f43cace\": rpc error: code = NotFound desc = could not find container \"c8f59d828eeb5a8c404015a365ddc7cdc9198215f9c1d95d2820a8af4f43cace\": container with ID starting with c8f59d828eeb5a8c404015a365ddc7cdc9198215f9c1d95d2820a8af4f43cace not found: ID does not exist"
Sep 11 18:07:05.068490 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:05.068458    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"]
Sep 11 18:07:05.079193 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:05.079161    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-7e61be66/e2e-lifecycle-model-7e61be66-kserve-79d8b858cf-wqbqh"]
Sep 11 18:07:07.053460 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:07.053412    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa59b260-9f5e-40a9-a6b6-37d54b3e550a" path="/var/lib/kubelet/pods/aa59b260-9f5e-40a9-a6b6-37d54b3e550a/volumes"
Sep 11 18:07:17.540980 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:17.540903    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:07:33.190659 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:33.190625    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:07:52.679011 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:52.678969    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:07:56.084120 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:56.084073    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7865-3bf851"]
Sep 11 18:07:56.084549 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:56.084479    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85a873b6-8f86-4acd-a8e1-b4151a856c43" containerName="payload-pre-processing"
Sep 11 18:07:56.084549 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:56.084494    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="85a873b6-8f86-4acd-a8e1-b4151a856c43" containerName="payload-pre-processing"
Sep 11 18:07:56.084549 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:56.084509    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa59b260-9f5e-40a9-a6b6-37d54b3e550a" containerName="main"
Sep 11 18:07:56.084549 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:56.084515    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa59b260-9f5e-40a9-a6b6-37d54b3e550a" containerName="main"
Sep 11 18:07:56.084549 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:56.084532    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="519955fa-6eef-40a5-ae6c-3fb307acdfb6" containerName="maas-api"
Sep 11 18:07:56.084549 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:56.084538    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="519955fa-6eef-40a5-ae6c-3fb307acdfb6" containerName="maas-api"
Sep 11 18:07:56.084743 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:56.084599    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="85a873b6-8f86-4acd-a8e1-b4151a856c43" containerName="payload-pre-processing"
Sep 11 18:07:56.084743 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:56.084609    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa59b260-9f5e-40a9-a6b6-37d54b3e550a" containerName="main"
Sep 11 18:07:56.084743 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:56.084615    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="519955fa-6eef-40a5-ae6c-3fb307acdfb6" containerName="maas-api"
Sep 11 18:07:56.088753 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:56.088736    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7865-3bf851"
Sep 11 18:07:56.092728 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:56.092707    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nbc6p\""
Sep 11 18:07:56.108605 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:56.108573    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7865-3bf851"]
Sep 11 18:07:56.175416 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:56.175372    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wbgc\" (UniqueName: \"kubernetes.io/projected/68036421-0e7f-4574-8e6c-45740a20cedd-kube-api-access-8wbgc\") pod \"test-curl-7865-3bf851\" (UID: \"68036421-0e7f-4574-8e6c-45740a20cedd\") " pod="opendatahub/test-curl-7865-3bf851"
Sep 11 18:07:56.276046 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:56.276006    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wbgc\" (UniqueName: \"kubernetes.io/projected/68036421-0e7f-4574-8e6c-45740a20cedd-kube-api-access-8wbgc\") pod \"test-curl-7865-3bf851\" (UID: \"68036421-0e7f-4574-8e6c-45740a20cedd\") " pod="opendatahub/test-curl-7865-3bf851"
Sep 11 18:07:56.287629 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:56.287598    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wbgc\" (UniqueName: \"kubernetes.io/projected/68036421-0e7f-4574-8e6c-45740a20cedd-kube-api-access-8wbgc\") pod \"test-curl-7865-3bf851\" (UID: \"68036421-0e7f-4574-8e6c-45740a20cedd\") " pod="opendatahub/test-curl-7865-3bf851"
Sep 11 18:07:56.397996 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:56.397955    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7865-3bf851"
Sep 11 18:07:56.685667 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:56.682812    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7865-3bf851"]
Sep 11 18:07:57.312690 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:57.312649    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7865-3bf851" event={"ID":"68036421-0e7f-4574-8e6c-45740a20cedd","Type":"ContainerStarted","Data":"58145153e67d202379091dfbf50370cb91088f33f30f6216de0dad601bf6787e"}
Sep 11 18:07:57.312690 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:57.312698    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7865-3bf851" event={"ID":"68036421-0e7f-4574-8e6c-45740a20cedd","Type":"ContainerStarted","Data":"0d11949f571121155579757d8ac9369aad4e3d02289125d38fd9840b6d64a0f3"}
Sep 11 18:07:57.349720 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:57.349659    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7865-3bf851" podStartSLOduration=1.349641308 podStartE2EDuration="1.349641308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:57.349065714 +0000 UTC m=+1382.794166286" watchObservedRunningTime="2026-09-11 18:07:57.349641308 +0000 UTC m=+1382.794741881"
Sep 11 18:07:58.296639 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:58.296545    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7865-3bf851"]
Sep 11 18:07:58.302647 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:58.302614    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7865-3bf851"]
Sep 11 18:07:58.319645 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:07:58.319613    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7865-3bf851" container="test-curl-7865-3bf851"
Sep 11 18:07:59.324490 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:07:59.324445    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7865-3bf851" podUID="68036421-0e7f-4574-8e6c-45740a20cedd" containerName="test-curl-7865-3bf851" containerID="cri-o://58145153e67d202379091dfbf50370cb91088f33f30f6216de0dad601bf6787e" gracePeriod=2
Sep 11 18:08:00.163183 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:00.163137    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"]
Sep 11 18:08:01.147111 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:01.147075    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"]
Sep 11 18:08:01.473941 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:01.473910    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7865-3bf851"
Sep 11 18:08:01.519035 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:01.519001    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wbgc\" (UniqueName: \"kubernetes.io/projected/68036421-0e7f-4574-8e6c-45740a20cedd-kube-api-access-8wbgc\") pod \"68036421-0e7f-4574-8e6c-45740a20cedd\" (UID: \"68036421-0e7f-4574-8e6c-45740a20cedd\") "
Sep 11 18:08:01.521165 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:01.521130    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68036421-0e7f-4574-8e6c-45740a20cedd-kube-api-access-8wbgc" (OuterVolumeSpecName: "kube-api-access-8wbgc") pod "68036421-0e7f-4574-8e6c-45740a20cedd" (UID: "68036421-0e7f-4574-8e6c-45740a20cedd"). InnerVolumeSpecName "kube-api-access-8wbgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:08:01.620423 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:01.620384    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wbgc\" (UniqueName: \"kubernetes.io/projected/68036421-0e7f-4574-8e6c-45740a20cedd-kube-api-access-8wbgc\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:08:02.344207 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:02.344174    2559 generic.go:356] "Generic (PLEG): container finished" podID="68036421-0e7f-4574-8e6c-45740a20cedd" containerID="58145153e67d202379091dfbf50370cb91088f33f30f6216de0dad601bf6787e" exitCode=137
Sep 11 18:08:02.344732 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:02.344227    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7865-3bf851"
Sep 11 18:08:02.344732 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:02.344272    2559 scope.go:117] "RemoveContainer" containerID="58145153e67d202379091dfbf50370cb91088f33f30f6216de0dad601bf6787e"
Sep 11 18:08:02.353716 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:02.353695    2559 scope.go:117] "RemoveContainer" containerID="58145153e67d202379091dfbf50370cb91088f33f30f6216de0dad601bf6787e"
Sep 11 18:08:02.354036 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:08:02.354015    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58145153e67d202379091dfbf50370cb91088f33f30f6216de0dad601bf6787e\": container with ID starting with 58145153e67d202379091dfbf50370cb91088f33f30f6216de0dad601bf6787e not found: ID does not exist" containerID="58145153e67d202379091dfbf50370cb91088f33f30f6216de0dad601bf6787e"
Sep 11 18:08:02.354104 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:02.354050    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58145153e67d202379091dfbf50370cb91088f33f30f6216de0dad601bf6787e"} err="failed to get container status \"58145153e67d202379091dfbf50370cb91088f33f30f6216de0dad601bf6787e\": rpc error: code = NotFound desc = could not find container \"58145153e67d202379091dfbf50370cb91088f33f30f6216de0dad601bf6787e\": container with ID starting with 58145153e67d202379091dfbf50370cb91088f33f30f6216de0dad601bf6787e not found: ID does not exist"
Sep 11 18:08:02.357534 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:08:02.357510    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7865-3bf851\" is forbidden: User \"system:node:ip-10-0-138-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-96.ec2.internal' and this object" podUID="68036421-0e7f-4574-8e6c-45740a20cedd" pod="opendatahub/test-curl-7865-3bf851"
Sep 11 18:08:03.052838 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:03.052790    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68036421-0e7f-4574-8e6c-45740a20cedd" path="/var/lib/kubelet/pods/68036421-0e7f-4574-8e6c-45740a20cedd/volumes"
Sep 11 18:08:09.600004 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.599965    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"]
Sep 11 18:08:09.600496 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.600337    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68036421-0e7f-4574-8e6c-45740a20cedd" containerName="test-curl-7865-3bf851"
Sep 11 18:08:09.600496 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.600365    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="68036421-0e7f-4574-8e6c-45740a20cedd" containerName="test-curl-7865-3bf851"
Sep 11 18:08:09.603338 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.600973    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="68036421-0e7f-4574-8e6c-45740a20cedd" containerName="test-curl-7865-3bf851"
Sep 11 18:08:09.609872 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.609845    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:08:09.612999 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.612974    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-386935\"/\"sub-test-model-386935-kserve-self-signed-certs\""
Sep 11 18:08:09.628310 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.628276    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"]
Sep 11 18:08:09.683337 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.683306    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff00d624-6d9b-4e80-bd6f-54343473e6e1-dshm\") pod \"sub-test-model-386935-kserve-995f54b75-mplbx\" (UID: \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\") " pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:08:09.683517 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.683342    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff00d624-6d9b-4e80-bd6f-54343473e6e1-model-cache\") pod \"sub-test-model-386935-kserve-995f54b75-mplbx\" (UID: \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\") " pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:08:09.683517 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.683440    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9kwt\" (UniqueName: \"kubernetes.io/projected/ff00d624-6d9b-4e80-bd6f-54343473e6e1-kube-api-access-p9kwt\") pod \"sub-test-model-386935-kserve-995f54b75-mplbx\" (UID: \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\") " pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:08:09.683517 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.683498    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff00d624-6d9b-4e80-bd6f-54343473e6e1-tmp-dir\") pod \"sub-test-model-386935-kserve-995f54b75-mplbx\" (UID: \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\") " pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:08:09.683627 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.683524    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff00d624-6d9b-4e80-bd6f-54343473e6e1-tls-certs\") pod \"sub-test-model-386935-kserve-995f54b75-mplbx\" (UID: \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\") " pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:08:09.683627 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.683553    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff00d624-6d9b-4e80-bd6f-54343473e6e1-home\") pod \"sub-test-model-386935-kserve-995f54b75-mplbx\" (UID: \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\") " pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:08:09.784149 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.784112    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9kwt\" (UniqueName: \"kubernetes.io/projected/ff00d624-6d9b-4e80-bd6f-54343473e6e1-kube-api-access-p9kwt\") pod \"sub-test-model-386935-kserve-995f54b75-mplbx\" (UID: \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\") " pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:08:09.784319 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.784169    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff00d624-6d9b-4e80-bd6f-54343473e6e1-tmp-dir\") pod \"sub-test-model-386935-kserve-995f54b75-mplbx\" (UID: \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\") " pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:08:09.784319 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.784259    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff00d624-6d9b-4e80-bd6f-54343473e6e1-tls-certs\") pod \"sub-test-model-386935-kserve-995f54b75-mplbx\" (UID: \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\") " pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:08:09.784319 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.784288    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff00d624-6d9b-4e80-bd6f-54343473e6e1-home\") pod \"sub-test-model-386935-kserve-995f54b75-mplbx\" (UID: \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\") " pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:08:09.784484 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.784464    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff00d624-6d9b-4e80-bd6f-54343473e6e1-dshm\") pod \"sub-test-model-386935-kserve-995f54b75-mplbx\" (UID: \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\") " pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:08:09.784521 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.784509    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff00d624-6d9b-4e80-bd6f-54343473e6e1-model-cache\") pod \"sub-test-model-386935-kserve-995f54b75-mplbx\" (UID: \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\") " pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:08:09.784713 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.784690    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff00d624-6d9b-4e80-bd6f-54343473e6e1-tmp-dir\") pod \"sub-test-model-386935-kserve-995f54b75-mplbx\" (UID: \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\") " pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:08:09.784758 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.784736    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff00d624-6d9b-4e80-bd6f-54343473e6e1-home\") pod \"sub-test-model-386935-kserve-995f54b75-mplbx\" (UID: \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\") " pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:08:09.784816 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.784802    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff00d624-6d9b-4e80-bd6f-54343473e6e1-model-cache\") pod \"sub-test-model-386935-kserve-995f54b75-mplbx\" (UID: \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\") " pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:08:09.786754 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.786729    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff00d624-6d9b-4e80-bd6f-54343473e6e1-dshm\") pod \"sub-test-model-386935-kserve-995f54b75-mplbx\" (UID: \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\") " pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:08:09.786984 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.786967    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff00d624-6d9b-4e80-bd6f-54343473e6e1-tls-certs\") pod \"sub-test-model-386935-kserve-995f54b75-mplbx\" (UID: \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\") " pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:08:09.801714 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.801686    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9kwt\" (UniqueName: \"kubernetes.io/projected/ff00d624-6d9b-4e80-bd6f-54343473e6e1-kube-api-access-p9kwt\") pod \"sub-test-model-386935-kserve-995f54b75-mplbx\" (UID: \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\") " pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:08:09.919853 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:09.919811    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:08:10.219641 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:10.219613    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"]
Sep 11 18:08:10.221850 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:08:10.221825    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff00d624_6d9b_4e80_bd6f_54343473e6e1.slice/crio-b8b67511854cad2ede3434baa5d434f7b3311cd5b68d797f9a299696705476ec WatchSource:0}: Error finding container b8b67511854cad2ede3434baa5d434f7b3311cd5b68d797f9a299696705476ec: Status 404 returned error can't find the container with id b8b67511854cad2ede3434baa5d434f7b3311cd5b68d797f9a299696705476ec
Sep 11 18:08:10.392869 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:10.392822    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx" event={"ID":"ff00d624-6d9b-4e80-bd6f-54343473e6e1","Type":"ContainerStarted","Data":"4c62ff35e111e4fdf642a7d1d2ec5aa4d07844fe765d5a1a3cd5060879f113a9"}
Sep 11 18:08:10.393063 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:10.392878    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx" event={"ID":"ff00d624-6d9b-4e80-bd6f-54343473e6e1","Type":"ContainerStarted","Data":"b8b67511854cad2ede3434baa5d434f7b3311cd5b68d797f9a299696705476ec"}
Sep 11 18:08:10.425553 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:10.425498    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx" podStartSLOduration=1.425481263 podStartE2EDuration="1.425481263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:08:10.42451767 +0000 UTC m=+1395.869618242" watchObservedRunningTime="2026-09-11 18:08:10.425481263 +0000 UTC m=+1395.870581835"
Sep 11 18:08:13.454202 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:13.454163    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:08:15.183535 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:15.183486    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf" podUID="7174b0e2-83f4-4ebf-bb60-6f7f3a93015b" containerName="main" containerID="cri-o://9c1e11dfeb41e6bdd85d995c0f06aa91f05aaae9afaaea4452cb4dd2fd75e8b5" gracePeriod=45
Sep 11 18:08:15.428299 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:15.428259    2559 generic.go:356] "Generic (PLEG): container finished" podID="7174b0e2-83f4-4ebf-bb60-6f7f3a93015b" containerID="9c1e11dfeb41e6bdd85d995c0f06aa91f05aaae9afaaea4452cb4dd2fd75e8b5" exitCode=0
Sep 11 18:08:15.428443 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:15.428339    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf" event={"ID":"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b","Type":"ContainerDied","Data":"9c1e11dfeb41e6bdd85d995c0f06aa91f05aaae9afaaea4452cb4dd2fd75e8b5"}
Sep 11 18:08:15.452238 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:15.452203    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:08:15.541807 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:15.541762    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-tmp-dir\") pod \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\" (UID: \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\") "
Sep 11 18:08:15.542010 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:15.541848    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-home\") pod \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\" (UID: \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\") "
Sep 11 18:08:15.542010 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:15.541904    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-model-cache\") pod \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\" (UID: \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\") "
Sep 11 18:08:15.542010 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:15.541941    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pkgc\" (UniqueName: \"kubernetes.io/projected/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-kube-api-access-4pkgc\") pod \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\" (UID: \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\") "
Sep 11 18:08:15.542185 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:15.542012    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-dshm\") pod \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\" (UID: \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\") "
Sep 11 18:08:15.542185 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:15.542063    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-tls-certs\") pod \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\" (UID: \"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b\") "
Sep 11 18:08:15.542185 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:15.542049    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7174b0e2-83f4-4ebf-bb60-6f7f3a93015b" (UID: "7174b0e2-83f4-4ebf-bb60-6f7f3a93015b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:08:15.542185 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:15.542123    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-home" (OuterVolumeSpecName: "home") pod "7174b0e2-83f4-4ebf-bb60-6f7f3a93015b" (UID: "7174b0e2-83f4-4ebf-bb60-6f7f3a93015b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:08:15.542185 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:15.542161    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-model-cache" (OuterVolumeSpecName: "model-cache") pod "7174b0e2-83f4-4ebf-bb60-6f7f3a93015b" (UID: "7174b0e2-83f4-4ebf-bb60-6f7f3a93015b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:08:15.542482 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:15.542333    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-tmp-dir\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:08:15.542482 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:15.542372    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-home\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:08:15.542482 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:15.542388    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-model-cache\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:08:15.544345 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:15.544321    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7174b0e2-83f4-4ebf-bb60-6f7f3a93015b" (UID: "7174b0e2-83f4-4ebf-bb60-6f7f3a93015b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:08:15.544444 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:15.544340    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-dshm" (OuterVolumeSpecName: "dshm") pod "7174b0e2-83f4-4ebf-bb60-6f7f3a93015b" (UID: "7174b0e2-83f4-4ebf-bb60-6f7f3a93015b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:08:15.544444 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:15.544383    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-kube-api-access-4pkgc" (OuterVolumeSpecName: "kube-api-access-4pkgc") pod "7174b0e2-83f4-4ebf-bb60-6f7f3a93015b" (UID: "7174b0e2-83f4-4ebf-bb60-6f7f3a93015b"). InnerVolumeSpecName "kube-api-access-4pkgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:08:15.643055 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:15.643017    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pkgc\" (UniqueName: \"kubernetes.io/projected/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-kube-api-access-4pkgc\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:08:15.643055 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:15.643050    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-dshm\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:08:15.643055 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:15.643061    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b-tls-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:08:16.166622 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.166579    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w" podUID="81016762-cd02-4fda-a227-5b191dfd4d1d" containerName="main" containerID="cri-o://b9907f9bacff9fc2df9bd08916de917c26f30942d6df95852a1b108dee87bc88" gracePeriod=45
Sep 11 18:08:16.419012 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.418941    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:08:16.435455 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.435420    2559 generic.go:356] "Generic (PLEG): container finished" podID="81016762-cd02-4fda-a227-5b191dfd4d1d" containerID="b9907f9bacff9fc2df9bd08916de917c26f30942d6df95852a1b108dee87bc88" exitCode=0
Sep 11 18:08:16.435643 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.435473    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"
Sep 11 18:08:16.435643 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.435500    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w" event={"ID":"81016762-cd02-4fda-a227-5b191dfd4d1d","Type":"ContainerDied","Data":"b9907f9bacff9fc2df9bd08916de917c26f30942d6df95852a1b108dee87bc88"}
Sep 11 18:08:16.435643 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.435555    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w" event={"ID":"81016762-cd02-4fda-a227-5b191dfd4d1d","Type":"ContainerDied","Data":"606b07e17d3fa4efc90116f433022f4e89df21759503d6f6b77ff4eb68862328"}
Sep 11 18:08:16.435643 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.435578    2559 scope.go:117] "RemoveContainer" containerID="b9907f9bacff9fc2df9bd08916de917c26f30942d6df95852a1b108dee87bc88"
Sep 11 18:08:16.436979 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.436955    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf" event={"ID":"7174b0e2-83f4-4ebf-bb60-6f7f3a93015b","Type":"ContainerDied","Data":"76411924a71bd743df337cf87965501ddd958a119686eb2a53a0c74cd1f0a2dd"}
Sep 11 18:08:16.437065 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.436963    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"
Sep 11 18:08:16.447656 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.447631    2559 scope.go:117] "RemoveContainer" containerID="b9907f9bacff9fc2df9bd08916de917c26f30942d6df95852a1b108dee87bc88"
Sep 11 18:08:16.447980 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:08:16.447959    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9907f9bacff9fc2df9bd08916de917c26f30942d6df95852a1b108dee87bc88\": container with ID starting with b9907f9bacff9fc2df9bd08916de917c26f30942d6df95852a1b108dee87bc88 not found: ID does not exist" containerID="b9907f9bacff9fc2df9bd08916de917c26f30942d6df95852a1b108dee87bc88"
Sep 11 18:08:16.448054 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.447987    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9907f9bacff9fc2df9bd08916de917c26f30942d6df95852a1b108dee87bc88"} err="failed to get container status \"b9907f9bacff9fc2df9bd08916de917c26f30942d6df95852a1b108dee87bc88\": rpc error: code = NotFound desc = could not find container \"b9907f9bacff9fc2df9bd08916de917c26f30942d6df95852a1b108dee87bc88\": container with ID starting with b9907f9bacff9fc2df9bd08916de917c26f30942d6df95852a1b108dee87bc88 not found: ID does not exist"
Sep 11 18:08:16.448054 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.448003    2559 scope.go:117] "RemoveContainer" containerID="9c1e11dfeb41e6bdd85d995c0f06aa91f05aaae9afaaea4452cb4dd2fd75e8b5"
Sep 11 18:08:16.451447 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.451420    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7nzs\" (UniqueName: \"kubernetes.io/projected/81016762-cd02-4fda-a227-5b191dfd4d1d-kube-api-access-s7nzs\") pod \"81016762-cd02-4fda-a227-5b191dfd4d1d\" (UID: \"81016762-cd02-4fda-a227-5b191dfd4d1d\") "
Sep 11 18:08:16.451563 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.451503    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81016762-cd02-4fda-a227-5b191dfd4d1d-tls-certs\") pod \"81016762-cd02-4fda-a227-5b191dfd4d1d\" (UID: \"81016762-cd02-4fda-a227-5b191dfd4d1d\") "
Sep 11 18:08:16.451637 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.451577    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81016762-cd02-4fda-a227-5b191dfd4d1d-tmp-dir\") pod \"81016762-cd02-4fda-a227-5b191dfd4d1d\" (UID: \"81016762-cd02-4fda-a227-5b191dfd4d1d\") "
Sep 11 18:08:16.451637 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.451604    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81016762-cd02-4fda-a227-5b191dfd4d1d-home\") pod \"81016762-cd02-4fda-a227-5b191dfd4d1d\" (UID: \"81016762-cd02-4fda-a227-5b191dfd4d1d\") "
Sep 11 18:08:16.451779 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.451652    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81016762-cd02-4fda-a227-5b191dfd4d1d-model-cache\") pod \"81016762-cd02-4fda-a227-5b191dfd4d1d\" (UID: \"81016762-cd02-4fda-a227-5b191dfd4d1d\") "
Sep 11 18:08:16.451779 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.451682    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81016762-cd02-4fda-a227-5b191dfd4d1d-dshm\") pod \"81016762-cd02-4fda-a227-5b191dfd4d1d\" (UID: \"81016762-cd02-4fda-a227-5b191dfd4d1d\") "
Sep 11 18:08:16.451890 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.451845    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81016762-cd02-4fda-a227-5b191dfd4d1d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "81016762-cd02-4fda-a227-5b191dfd4d1d" (UID: "81016762-cd02-4fda-a227-5b191dfd4d1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:08:16.451952 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.451885    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81016762-cd02-4fda-a227-5b191dfd4d1d-home" (OuterVolumeSpecName: "home") pod "81016762-cd02-4fda-a227-5b191dfd4d1d" (UID: "81016762-cd02-4fda-a227-5b191dfd4d1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:08:16.452012 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.451961    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81016762-cd02-4fda-a227-5b191dfd4d1d-model-cache" (OuterVolumeSpecName: "model-cache") pod "81016762-cd02-4fda-a227-5b191dfd4d1d" (UID: "81016762-cd02-4fda-a227-5b191dfd4d1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:08:16.452111 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.452088    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81016762-cd02-4fda-a227-5b191dfd4d1d-tmp-dir\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:08:16.452183 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.452114    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81016762-cd02-4fda-a227-5b191dfd4d1d-home\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:08:16.452183 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.452125    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81016762-cd02-4fda-a227-5b191dfd4d1d-model-cache\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:08:16.454215 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.454182    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81016762-cd02-4fda-a227-5b191dfd4d1d-dshm" (OuterVolumeSpecName: "dshm") pod "81016762-cd02-4fda-a227-5b191dfd4d1d" (UID: "81016762-cd02-4fda-a227-5b191dfd4d1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:08:16.454306 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.454275    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81016762-cd02-4fda-a227-5b191dfd4d1d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81016762-cd02-4fda-a227-5b191dfd4d1d" (UID: "81016762-cd02-4fda-a227-5b191dfd4d1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:08:16.454306 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.454285    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81016762-cd02-4fda-a227-5b191dfd4d1d-kube-api-access-s7nzs" (OuterVolumeSpecName: "kube-api-access-s7nzs") pod "81016762-cd02-4fda-a227-5b191dfd4d1d" (UID: "81016762-cd02-4fda-a227-5b191dfd4d1d"). InnerVolumeSpecName "kube-api-access-s7nzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:08:16.465919 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.465889    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"]
Sep 11 18:08:16.480473 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.480441    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-386935/auth-test-model-386935-kserve-69dd649579-bbjkf"]
Sep 11 18:08:16.552906 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.552875    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81016762-cd02-4fda-a227-5b191dfd4d1d-dshm\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:08:16.552906 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.552906    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7nzs\" (UniqueName: \"kubernetes.io/projected/81016762-cd02-4fda-a227-5b191dfd4d1d-kube-api-access-s7nzs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:08:16.552906 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.552917    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81016762-cd02-4fda-a227-5b191dfd4d1d-tls-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:08:16.770836 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.770796    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"]
Sep 11 18:08:16.779994 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:16.779953    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ebea5/auth-test-model-3ebea5-kserve-f5788cddc-mg96w"]
Sep 11 18:08:17.053226 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:17.053147    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7174b0e2-83f4-4ebf-bb60-6f7f3a93015b" path="/var/lib/kubelet/pods/7174b0e2-83f4-4ebf-bb60-6f7f3a93015b/volumes"
Sep 11 18:08:17.053605 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:17.053591    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81016762-cd02-4fda-a227-5b191dfd4d1d" path="/var/lib/kubelet/pods/81016762-cd02-4fda-a227-5b191dfd4d1d/volumes"
Sep 11 18:08:19.919947 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:19.919895    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:08:19.920441 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:19.919981    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:08:19.932981 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:19.932949    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:08:20.473507 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:20.473476    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:08:35.006902 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.006865    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"]
Sep 11 18:08:35.007292 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.007258    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7174b0e2-83f4-4ebf-bb60-6f7f3a93015b" containerName="main"
Sep 11 18:08:35.007292 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.007270    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7174b0e2-83f4-4ebf-bb60-6f7f3a93015b" containerName="main"
Sep 11 18:08:35.007408 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.007300    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81016762-cd02-4fda-a227-5b191dfd4d1d" containerName="main"
Sep 11 18:08:35.007408 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.007306    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="81016762-cd02-4fda-a227-5b191dfd4d1d" containerName="main"
Sep 11 18:08:35.007484 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.007426    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7174b0e2-83f4-4ebf-bb60-6f7f3a93015b" containerName="main"
Sep 11 18:08:35.007484 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.007437    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="81016762-cd02-4fda-a227-5b191dfd4d1d" containerName="main"
Sep 11 18:08:35.010476 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.010459    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:08:35.014596 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.014573    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3ebea5\"/\"default-dockercfg-5wbch\""
Sep 11 18:08:35.014840 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.014821    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3ebea5\"/\"openshift-service-ca.crt\""
Sep 11 18:08:35.015607 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.015588    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3ebea5\"/\"sub-test-model-3ebea5-kserve-self-signed-certs\""
Sep 11 18:08:35.015869 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.015846    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3ebea5\"/\"kube-root-ca.crt\""
Sep 11 18:08:35.030716 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.030687    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"]
Sep 11 18:08:35.130203 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.130159    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a871d8a2-a8ed-44e8-8583-88739f7b1b36-model-cache\") pod \"sub-test-model-3ebea5-kserve-7869bb844f-fvnt7\" (UID: \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:08:35.130203 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.130205    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a871d8a2-a8ed-44e8-8583-88739f7b1b36-tmp-dir\") pod \"sub-test-model-3ebea5-kserve-7869bb844f-fvnt7\" (UID: \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:08:35.130542 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.130259    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7lcc\" (UniqueName: \"kubernetes.io/projected/a871d8a2-a8ed-44e8-8583-88739f7b1b36-kube-api-access-s7lcc\") pod \"sub-test-model-3ebea5-kserve-7869bb844f-fvnt7\" (UID: \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:08:35.130542 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.130325    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a871d8a2-a8ed-44e8-8583-88739f7b1b36-home\") pod \"sub-test-model-3ebea5-kserve-7869bb844f-fvnt7\" (UID: \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:08:35.130542 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.130472    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a871d8a2-a8ed-44e8-8583-88739f7b1b36-tls-certs\") pod \"sub-test-model-3ebea5-kserve-7869bb844f-fvnt7\" (UID: \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:08:35.130542 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.130522    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a871d8a2-a8ed-44e8-8583-88739f7b1b36-dshm\") pod \"sub-test-model-3ebea5-kserve-7869bb844f-fvnt7\" (UID: \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:08:35.231574 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.231529    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a871d8a2-a8ed-44e8-8583-88739f7b1b36-tls-certs\") pod \"sub-test-model-3ebea5-kserve-7869bb844f-fvnt7\" (UID: \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:08:35.231788 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.231586    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a871d8a2-a8ed-44e8-8583-88739f7b1b36-dshm\") pod \"sub-test-model-3ebea5-kserve-7869bb844f-fvnt7\" (UID: \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:08:35.231788 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.231624    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a871d8a2-a8ed-44e8-8583-88739f7b1b36-model-cache\") pod \"sub-test-model-3ebea5-kserve-7869bb844f-fvnt7\" (UID: \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:08:35.231788 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.231649    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a871d8a2-a8ed-44e8-8583-88739f7b1b36-tmp-dir\") pod \"sub-test-model-3ebea5-kserve-7869bb844f-fvnt7\" (UID: \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:08:35.231788 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.231677    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7lcc\" (UniqueName: \"kubernetes.io/projected/a871d8a2-a8ed-44e8-8583-88739f7b1b36-kube-api-access-s7lcc\") pod \"sub-test-model-3ebea5-kserve-7869bb844f-fvnt7\" (UID: \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:08:35.231788 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.231756    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a871d8a2-a8ed-44e8-8583-88739f7b1b36-home\") pod \"sub-test-model-3ebea5-kserve-7869bb844f-fvnt7\" (UID: \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:08:35.232111 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.232090    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a871d8a2-a8ed-44e8-8583-88739f7b1b36-model-cache\") pod \"sub-test-model-3ebea5-kserve-7869bb844f-fvnt7\" (UID: \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:08:35.232185 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.232114    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a871d8a2-a8ed-44e8-8583-88739f7b1b36-tmp-dir\") pod \"sub-test-model-3ebea5-kserve-7869bb844f-fvnt7\" (UID: \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:08:35.232248 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.232227    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a871d8a2-a8ed-44e8-8583-88739f7b1b36-home\") pod \"sub-test-model-3ebea5-kserve-7869bb844f-fvnt7\" (UID: \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:08:35.234098 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.234076    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a871d8a2-a8ed-44e8-8583-88739f7b1b36-dshm\") pod \"sub-test-model-3ebea5-kserve-7869bb844f-fvnt7\" (UID: \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:08:35.234280 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.234262    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a871d8a2-a8ed-44e8-8583-88739f7b1b36-tls-certs\") pod \"sub-test-model-3ebea5-kserve-7869bb844f-fvnt7\" (UID: \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:08:35.242986 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.242965    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7lcc\" (UniqueName: \"kubernetes.io/projected/a871d8a2-a8ed-44e8-8583-88739f7b1b36-kube-api-access-s7lcc\") pod \"sub-test-model-3ebea5-kserve-7869bb844f-fvnt7\" (UID: \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:08:35.319527 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.319417    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:08:35.608464 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:35.608377    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"]
Sep 11 18:08:35.611485 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:08:35.611454    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda871d8a2_a8ed_44e8_8583_88739f7b1b36.slice/crio-434b98b8e4f180c2881a80e3c39a334a69228320065d4ba149eaec48c88f6d68 WatchSource:0}: Error finding container 434b98b8e4f180c2881a80e3c39a334a69228320065d4ba149eaec48c88f6d68: Status 404 returned error can't find the container with id 434b98b8e4f180c2881a80e3c39a334a69228320065d4ba149eaec48c88f6d68
Sep 11 18:08:36.562039 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:36.562003    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7" event={"ID":"a871d8a2-a8ed-44e8-8583-88739f7b1b36","Type":"ContainerStarted","Data":"41dce4b05b3ed1f46397572e5682ae49559b80878e75479bb566f2b13d51f514"}
Sep 11 18:08:36.562039 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:36.562050    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7" event={"ID":"a871d8a2-a8ed-44e8-8583-88739f7b1b36","Type":"ContainerStarted","Data":"434b98b8e4f180c2881a80e3c39a334a69228320065d4ba149eaec48c88f6d68"}
Sep 11 18:08:36.591939 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:36.591881    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7" podStartSLOduration=2.591865022 podStartE2EDuration="2.591865022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:08:36.590502028 +0000 UTC m=+1422.035602612" watchObservedRunningTime="2026-09-11 18:08:36.591865022 +0000 UTC m=+1422.036965593"
Sep 11 18:08:41.357567 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:41.357527    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:08:45.161560 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:45.161519    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6"]
Sep 11 18:08:45.166292 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:45.166266    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6"
Sep 11 18:08:45.171598 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:45.171573    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-6de04f-serving-cert\""
Sep 11 18:08:45.190605 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:45.190565    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6"]
Sep 11 18:08:45.320260 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:45.320217    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:08:45.320441 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:45.320281    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:08:45.333306 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:45.333260    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad247e53-5db0-4caa-8865-e4696eb756ff-openshift-service-ca\") pod \"maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6\" (UID: \"ad247e53-5db0-4caa-8865-e4696eb756ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6"
Sep 11 18:08:45.333513 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:45.333333    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:08:45.333513 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:45.333334    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad247e53-5db0-4caa-8865-e4696eb756ff-maas-api-tls\") pod \"maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6\" (UID: \"ad247e53-5db0-4caa-8865-e4696eb756ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6"
Sep 11 18:08:45.333513 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:45.333441    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad247e53-5db0-4caa-8865-e4696eb756ff-metrics-certs\") pod \"maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6\" (UID: \"ad247e53-5db0-4caa-8865-e4696eb756ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6"
Sep 11 18:08:45.333513 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:45.333465    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phmgp\" (UniqueName: \"kubernetes.io/projected/ad247e53-5db0-4caa-8865-e4696eb756ff-kube-api-access-phmgp\") pod \"maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6\" (UID: \"ad247e53-5db0-4caa-8865-e4696eb756ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6"
Sep 11 18:08:45.438340 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:45.436964    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad247e53-5db0-4caa-8865-e4696eb756ff-metrics-certs\") pod \"maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6\" (UID: \"ad247e53-5db0-4caa-8865-e4696eb756ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6"
Sep 11 18:08:45.438340 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:45.437024    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phmgp\" (UniqueName: \"kubernetes.io/projected/ad247e53-5db0-4caa-8865-e4696eb756ff-kube-api-access-phmgp\") pod \"maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6\" (UID: \"ad247e53-5db0-4caa-8865-e4696eb756ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6"
Sep 11 18:08:45.438340 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:45.437133    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad247e53-5db0-4caa-8865-e4696eb756ff-openshift-service-ca\") pod \"maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6\" (UID: \"ad247e53-5db0-4caa-8865-e4696eb756ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6"
Sep 11 18:08:45.438340 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:45.437217    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad247e53-5db0-4caa-8865-e4696eb756ff-maas-api-tls\") pod \"maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6\" (UID: \"ad247e53-5db0-4caa-8865-e4696eb756ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6"
Sep 11 18:08:45.441548 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:45.441514    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad247e53-5db0-4caa-8865-e4696eb756ff-openshift-service-ca\") pod \"maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6\" (UID: \"ad247e53-5db0-4caa-8865-e4696eb756ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6"
Sep 11 18:08:45.443584 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:45.443551    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad247e53-5db0-4caa-8865-e4696eb756ff-metrics-certs\") pod \"maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6\" (UID: \"ad247e53-5db0-4caa-8865-e4696eb756ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6"
Sep 11 18:08:45.444383 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:45.444339    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad247e53-5db0-4caa-8865-e4696eb756ff-maas-api-tls\") pod \"maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6\" (UID: \"ad247e53-5db0-4caa-8865-e4696eb756ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6"
Sep 11 18:08:45.456253 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:45.456222    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phmgp\" (UniqueName: \"kubernetes.io/projected/ad247e53-5db0-4caa-8865-e4696eb756ff-kube-api-access-phmgp\") pod \"maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6\" (UID: \"ad247e53-5db0-4caa-8865-e4696eb756ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6"
Sep 11 18:08:45.479865 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:45.479824    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6"
Sep 11 18:08:45.629943 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:45.629910    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:08:45.779209 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:45.779110    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6"]
Sep 11 18:08:45.783985 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:08:45.783953    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad247e53_5db0_4caa_8865_e4696eb756ff.slice/crio-944f50eb6e2d342bd91902e999e92cb7d902c5dc6bc9ff3436e318da1a5e630d WatchSource:0}: Error finding container 944f50eb6e2d342bd91902e999e92cb7d902c5dc6bc9ff3436e318da1a5e630d: Status 404 returned error can't find the container with id 944f50eb6e2d342bd91902e999e92cb7d902c5dc6bc9ff3436e318da1a5e630d
Sep 11 18:08:46.626765 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:46.626658    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6" event={"ID":"ad247e53-5db0-4caa-8865-e4696eb756ff","Type":"ContainerStarted","Data":"39c1508f463d5af3e5eeb2b1e65d0caff5ec0c2735287f8a0e3fe266dbe5ab2c"}
Sep 11 18:08:46.627274 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:46.626879    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6"
Sep 11 18:08:46.627274 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:46.626903    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6" event={"ID":"ad247e53-5db0-4caa-8865-e4696eb756ff","Type":"ContainerStarted","Data":"944f50eb6e2d342bd91902e999e92cb7d902c5dc6bc9ff3436e318da1a5e630d"}
Sep 11 18:08:46.709716 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:46.709658    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6" podStartSLOduration=1.325923437 podStartE2EDuration="1.709639888s" totalImagesPullingTime="383.716451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:08:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:08:46.707389817 +0000 UTC m=+1432.152490391" watchObservedRunningTime="2026-09-11 18:08:46.709639888 +0000 UTC m=+1432.154740511"
Sep 11 18:08:47.003483 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.003445    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw"]
Sep 11 18:08:47.007189 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.007169    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw"
Sep 11 18:08:47.010506 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.010478    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6de04f-dockercfg-mxwl7\""
Sep 11 18:08:47.010881 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.010861    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6de04f\""
Sep 11 18:08:47.031887 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.031855    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw"]
Sep 11 18:08:47.154565 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.154475    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6ccef1-d0e9-4e74-9cd1-cf7ac841c151-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw\" (UID: \"da6ccef1-d0e9-4e74-9cd1-cf7ac841c151\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw"
Sep 11 18:08:47.154731 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.154605    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7qc2\" (UniqueName: \"kubernetes.io/projected/da6ccef1-d0e9-4e74-9cd1-cf7ac841c151-kube-api-access-n7qc2\") pod \"payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw\" (UID: \"da6ccef1-d0e9-4e74-9cd1-cf7ac841c151\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw"
Sep 11 18:08:47.195240 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.195200    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5"]
Sep 11 18:08:47.198986 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.198960    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5"
Sep 11 18:08:47.213002 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.212970    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5"]
Sep 11 18:08:47.255469 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.255340    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7qc2\" (UniqueName: \"kubernetes.io/projected/da6ccef1-d0e9-4e74-9cd1-cf7ac841c151-kube-api-access-n7qc2\") pod \"payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw\" (UID: \"da6ccef1-d0e9-4e74-9cd1-cf7ac841c151\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw"
Sep 11 18:08:47.255469 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.255454    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6ccef1-d0e9-4e74-9cd1-cf7ac841c151-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw\" (UID: \"da6ccef1-d0e9-4e74-9cd1-cf7ac841c151\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw"
Sep 11 18:08:47.256188 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.256168    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6ccef1-d0e9-4e74-9cd1-cf7ac841c151-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw\" (UID: \"da6ccef1-d0e9-4e74-9cd1-cf7ac841c151\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw"
Sep 11 18:08:47.270695 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.270663    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7qc2\" (UniqueName: \"kubernetes.io/projected/da6ccef1-d0e9-4e74-9cd1-cf7ac841c151-kube-api-access-n7qc2\") pod \"payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw\" (UID: \"da6ccef1-d0e9-4e74-9cd1-cf7ac841c151\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw"
Sep 11 18:08:47.318136 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.318093    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw"
Sep 11 18:08:47.356485 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.356446    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d7a8e0f-f34c-4fb2-aae2-c8d16b118916-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5\" (UID: \"0d7a8e0f-f34c-4fb2-aae2-c8d16b118916\") " pod="openshift-ingress/payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5"
Sep 11 18:08:47.356669 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.356506    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5658r\" (UniqueName: \"kubernetes.io/projected/0d7a8e0f-f34c-4fb2-aae2-c8d16b118916-kube-api-access-5658r\") pod \"payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5\" (UID: \"0d7a8e0f-f34c-4fb2-aae2-c8d16b118916\") " pod="openshift-ingress/payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5"
Sep 11 18:08:47.457752 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.457716    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5658r\" (UniqueName: \"kubernetes.io/projected/0d7a8e0f-f34c-4fb2-aae2-c8d16b118916-kube-api-access-5658r\") pod \"payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5\" (UID: \"0d7a8e0f-f34c-4fb2-aae2-c8d16b118916\") " pod="openshift-ingress/payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5"
Sep 11 18:08:47.457967 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.457871    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d7a8e0f-f34c-4fb2-aae2-c8d16b118916-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5\" (UID: \"0d7a8e0f-f34c-4fb2-aae2-c8d16b118916\") " pod="openshift-ingress/payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5"
Sep 11 18:08:47.458619 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.458597    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d7a8e0f-f34c-4fb2-aae2-c8d16b118916-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5\" (UID: \"0d7a8e0f-f34c-4fb2-aae2-c8d16b118916\") " pod="openshift-ingress/payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5"
Sep 11 18:08:47.473276 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.473251    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5658r\" (UniqueName: \"kubernetes.io/projected/0d7a8e0f-f34c-4fb2-aae2-c8d16b118916-kube-api-access-5658r\") pod \"payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5\" (UID: \"0d7a8e0f-f34c-4fb2-aae2-c8d16b118916\") " pod="openshift-ingress/payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5"
Sep 11 18:08:47.509924 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.509827    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5"
Sep 11 18:08:47.617998 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.617966    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw"]
Sep 11 18:08:47.619664 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:08:47.619635    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6ccef1_d0e9_4e74_9cd1_cf7ac841c151.slice/crio-1f6d7b3f509cd0e0af7f38789381b70e6a5032072742b6775136f83fe051f934 WatchSource:0}: Error finding container 1f6d7b3f509cd0e0af7f38789381b70e6a5032072742b6775136f83fe051f934: Status 404 returned error can't find the container with id 1f6d7b3f509cd0e0af7f38789381b70e6a5032072742b6775136f83fe051f934
Sep 11 18:08:47.633058 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.633028    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw" event={"ID":"da6ccef1-d0e9-4e74-9cd1-cf7ac841c151","Type":"ContainerStarted","Data":"1f6d7b3f509cd0e0af7f38789381b70e6a5032072742b6775136f83fe051f934"}
Sep 11 18:08:47.811671 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:47.811640    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5"]
Sep 11 18:08:47.813454 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:08:47.813428    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d7a8e0f_f34c_4fb2_aae2_c8d16b118916.slice/crio-3b56d945d4ed2c47bc4421ece5d76fdd1e0b8ef5f94e1b3d2951c8ada32dcebc WatchSource:0}: Error finding container 3b56d945d4ed2c47bc4421ece5d76fdd1e0b8ef5f94e1b3d2951c8ada32dcebc: Status 404 returned error can't find the container with id 3b56d945d4ed2c47bc4421ece5d76fdd1e0b8ef5f94e1b3d2951c8ada32dcebc
Sep 11 18:08:48.641374 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:48.641305    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5" event={"ID":"0d7a8e0f-f34c-4fb2-aae2-c8d16b118916","Type":"ContainerStarted","Data":"7139259763b46d9867bb2f7f402ba9c6d0279a8ac74c81a845eb2d1c103fa665"}
Sep 11 18:08:48.641879 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:48.641379    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5" event={"ID":"0d7a8e0f-f34c-4fb2-aae2-c8d16b118916","Type":"ContainerStarted","Data":"3b56d945d4ed2c47bc4421ece5d76fdd1e0b8ef5f94e1b3d2951c8ada32dcebc"}
Sep 11 18:08:48.641879 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:48.641412    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5"
Sep 11 18:08:48.643104 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:48.643076    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw" event={"ID":"da6ccef1-d0e9-4e74-9cd1-cf7ac841c151","Type":"ContainerStarted","Data":"734351ed5870b1b28f4395250f36cb41dcc29529808155ae554597396ca7d159"}
Sep 11 18:08:48.643229 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:48.643167    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw"
Sep 11 18:08:48.721196 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:48.721136    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw" podStartSLOduration=2.721118856 podStartE2EDuration="2.721118856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:08:48.720503388 +0000 UTC m=+1434.165603959" watchObservedRunningTime="2026-09-11 18:08:48.721118856 +0000 UTC m=+1434.166219427"
Sep 11 18:08:48.722056 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:48.722014    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5" podStartSLOduration=1.722003837 podStartE2EDuration="1.722003837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:08:48.676670711 +0000 UTC m=+1434.121771282" watchObservedRunningTime="2026-09-11 18:08:48.722003837 +0000 UTC m=+1434.167104407"
Sep 11 18:08:52.638487 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:52.638450    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6"
Sep 11 18:08:59.651043 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:59.651009    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5"
Sep 11 18:08:59.651512 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:08:59.651168    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw"
Sep 11 18:09:03.363797 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:03.363754    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:09:20.927868 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:20.927828    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v"]
Sep 11 18:09:20.932709 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:20.932687    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v"
Sep 11 18:09:20.936904 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:20.936872    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f1705a\""
Sep 11 18:09:20.937068 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:20.936996    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f1705a-dockercfg-kd5vf\""
Sep 11 18:09:20.960286 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:20.960250    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v"]
Sep 11 18:09:21.104571 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:21.104539    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472fa464-1370-4d6d-a851-dd1159a7708d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v\" (UID: \"472fa464-1370-4d6d-a851-dd1159a7708d\") " pod="openshift-ingress/payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v"
Sep 11 18:09:21.104571 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:21.104572    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqdnf\" (UniqueName: \"kubernetes.io/projected/472fa464-1370-4d6d-a851-dd1159a7708d-kube-api-access-qqdnf\") pod \"payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v\" (UID: \"472fa464-1370-4d6d-a851-dd1159a7708d\") " pod="openshift-ingress/payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v"
Sep 11 18:09:21.206068 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:21.205965    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472fa464-1370-4d6d-a851-dd1159a7708d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v\" (UID: \"472fa464-1370-4d6d-a851-dd1159a7708d\") " pod="openshift-ingress/payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v"
Sep 11 18:09:21.206068 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:21.206018    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqdnf\" (UniqueName: \"kubernetes.io/projected/472fa464-1370-4d6d-a851-dd1159a7708d-kube-api-access-qqdnf\") pod \"payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v\" (UID: \"472fa464-1370-4d6d-a851-dd1159a7708d\") " pod="openshift-ingress/payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v"
Sep 11 18:09:21.206695 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:21.206674    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472fa464-1370-4d6d-a851-dd1159a7708d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v\" (UID: \"472fa464-1370-4d6d-a851-dd1159a7708d\") " pod="openshift-ingress/payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v"
Sep 11 18:09:21.216812 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:21.216792    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqdnf\" (UniqueName: \"kubernetes.io/projected/472fa464-1370-4d6d-a851-dd1159a7708d-kube-api-access-qqdnf\") pod \"payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v\" (UID: \"472fa464-1370-4d6d-a851-dd1159a7708d\") " pod="openshift-ingress/payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v"
Sep 11 18:09:21.243674 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:21.243632    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v"
Sep 11 18:09:21.539532 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:21.539403    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v"]
Sep 11 18:09:21.542532 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:09:21.542501    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod472fa464_1370_4d6d_a851_dd1159a7708d.slice/crio-b2553b285cccd365466cebd6becaa7f19aeafaf7ba8360daea2e04462006b7c5 WatchSource:0}: Error finding container b2553b285cccd365466cebd6becaa7f19aeafaf7ba8360daea2e04462006b7c5: Status 404 returned error can't find the container with id b2553b285cccd365466cebd6becaa7f19aeafaf7ba8360daea2e04462006b7c5
Sep 11 18:09:21.836449 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:21.836359    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v" event={"ID":"472fa464-1370-4d6d-a851-dd1159a7708d","Type":"ContainerStarted","Data":"b1f6b9490f63bd582f784b7a9fbf43e44766f8db411691cff383f67c226dcd89"}
Sep 11 18:09:21.836449 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:21.836415    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v" event={"ID":"472fa464-1370-4d6d-a851-dd1159a7708d","Type":"ContainerStarted","Data":"b2553b285cccd365466cebd6becaa7f19aeafaf7ba8360daea2e04462006b7c5"}
Sep 11 18:09:21.836643 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:21.836512    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v"
Sep 11 18:09:21.867465 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:21.867401    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v" podStartSLOduration=1.8673776530000001 podStartE2EDuration="1.867377653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:09:21.865917841 +0000 UTC m=+1467.311018412" watchObservedRunningTime="2026-09-11 18:09:21.867377653 +0000 UTC m=+1467.312478222"
Sep 11 18:09:31.725267 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:31.725228    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:09:32.846547 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:32.846515    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v"
Sep 11 18:09:41.865260 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:41.865222    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"]
Sep 11 18:09:42.768993 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:42.768956    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"]
Sep 11 18:09:54.497081 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:54.497042    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:09:56.884244 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:56.884184    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx" podUID="ff00d624-6d9b-4e80-bd6f-54343473e6e1" containerName="main" containerID="cri-o://4c62ff35e111e4fdf642a7d1d2ec5aa4d07844fe765d5a1a3cd5060879f113a9" gracePeriod=45
Sep 11 18:09:57.046998 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:57.046959    2559 generic.go:356] "Generic (PLEG): container finished" podID="ff00d624-6d9b-4e80-bd6f-54343473e6e1" containerID="4c62ff35e111e4fdf642a7d1d2ec5aa4d07844fe765d5a1a3cd5060879f113a9" exitCode=0
Sep 11 18:09:57.052360 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:57.052308    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx" event={"ID":"ff00d624-6d9b-4e80-bd6f-54343473e6e1","Type":"ContainerDied","Data":"4c62ff35e111e4fdf642a7d1d2ec5aa4d07844fe765d5a1a3cd5060879f113a9"}
Sep 11 18:09:57.137963 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:57.137898    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:09:57.156536 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:57.156495    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff00d624-6d9b-4e80-bd6f-54343473e6e1-tls-certs\") pod \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\" (UID: \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\") "
Sep 11 18:09:57.156748 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:57.156550    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff00d624-6d9b-4e80-bd6f-54343473e6e1-dshm\") pod \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\" (UID: \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\") "
Sep 11 18:09:57.156748 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:57.156606    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff00d624-6d9b-4e80-bd6f-54343473e6e1-home\") pod \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\" (UID: \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\") "
Sep 11 18:09:57.156748 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:57.156632    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff00d624-6d9b-4e80-bd6f-54343473e6e1-model-cache\") pod \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\" (UID: \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\") "
Sep 11 18:09:57.156748 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:57.156658    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff00d624-6d9b-4e80-bd6f-54343473e6e1-tmp-dir\") pod \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\" (UID: \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\") "
Sep 11 18:09:57.156748 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:57.156691    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9kwt\" (UniqueName: \"kubernetes.io/projected/ff00d624-6d9b-4e80-bd6f-54343473e6e1-kube-api-access-p9kwt\") pod \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\" (UID: \"ff00d624-6d9b-4e80-bd6f-54343473e6e1\") "
Sep 11 18:09:57.156997 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:57.156877    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff00d624-6d9b-4e80-bd6f-54343473e6e1-home" (OuterVolumeSpecName: "home") pod "ff00d624-6d9b-4e80-bd6f-54343473e6e1" (UID: "ff00d624-6d9b-4e80-bd6f-54343473e6e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:09:57.156997 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:57.156915    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff00d624-6d9b-4e80-bd6f-54343473e6e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "ff00d624-6d9b-4e80-bd6f-54343473e6e1" (UID: "ff00d624-6d9b-4e80-bd6f-54343473e6e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:09:57.157107 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:57.157085    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff00d624-6d9b-4e80-bd6f-54343473e6e1-home\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:09:57.157172 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:57.157114    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff00d624-6d9b-4e80-bd6f-54343473e6e1-model-cache\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:09:57.157172 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:57.157156    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff00d624-6d9b-4e80-bd6f-54343473e6e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ff00d624-6d9b-4e80-bd6f-54343473e6e1" (UID: "ff00d624-6d9b-4e80-bd6f-54343473e6e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:09:57.158804 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:57.158777    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff00d624-6d9b-4e80-bd6f-54343473e6e1-dshm" (OuterVolumeSpecName: "dshm") pod "ff00d624-6d9b-4e80-bd6f-54343473e6e1" (UID: "ff00d624-6d9b-4e80-bd6f-54343473e6e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:09:57.158920 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:57.158862    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff00d624-6d9b-4e80-bd6f-54343473e6e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff00d624-6d9b-4e80-bd6f-54343473e6e1" (UID: "ff00d624-6d9b-4e80-bd6f-54343473e6e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:09:57.158920 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:57.158896    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff00d624-6d9b-4e80-bd6f-54343473e6e1-kube-api-access-p9kwt" (OuterVolumeSpecName: "kube-api-access-p9kwt") pod "ff00d624-6d9b-4e80-bd6f-54343473e6e1" (UID: "ff00d624-6d9b-4e80-bd6f-54343473e6e1"). InnerVolumeSpecName "kube-api-access-p9kwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:09:57.257664 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:57.257623    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9kwt\" (UniqueName: \"kubernetes.io/projected/ff00d624-6d9b-4e80-bd6f-54343473e6e1-kube-api-access-p9kwt\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:09:57.257664 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:57.257661    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff00d624-6d9b-4e80-bd6f-54343473e6e1-tls-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:09:57.257664 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:57.257671    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff00d624-6d9b-4e80-bd6f-54343473e6e1-dshm\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:09:57.257664 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:57.257679    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff00d624-6d9b-4e80-bd6f-54343473e6e1-tmp-dir\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:09:57.788067 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:57.788006    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7" podUID="a871d8a2-a8ed-44e8-8583-88739f7b1b36" containerName="main" containerID="cri-o://41dce4b05b3ed1f46397572e5682ae49559b80878e75479bb566f2b13d51f514" gracePeriod=45
Sep 11 18:09:58.050970 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.050904    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:09:58.053086 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.053062    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx" event={"ID":"ff00d624-6d9b-4e80-bd6f-54343473e6e1","Type":"ContainerDied","Data":"b8b67511854cad2ede3434baa5d434f7b3311cd5b68d797f9a299696705476ec"}
Sep 11 18:09:58.053211 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.053102    2559 scope.go:117] "RemoveContainer" containerID="4c62ff35e111e4fdf642a7d1d2ec5aa4d07844fe765d5a1a3cd5060879f113a9"
Sep 11 18:09:58.053211 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.053070    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"
Sep 11 18:09:58.054490 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.054470    2559 generic.go:356] "Generic (PLEG): container finished" podID="a871d8a2-a8ed-44e8-8583-88739f7b1b36" containerID="41dce4b05b3ed1f46397572e5682ae49559b80878e75479bb566f2b13d51f514" exitCode=0
Sep 11 18:09:58.054582 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.054501    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"
Sep 11 18:09:58.054582 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.054558    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7" event={"ID":"a871d8a2-a8ed-44e8-8583-88739f7b1b36","Type":"ContainerDied","Data":"41dce4b05b3ed1f46397572e5682ae49559b80878e75479bb566f2b13d51f514"}
Sep 11 18:09:58.054698 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.054597    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7" event={"ID":"a871d8a2-a8ed-44e8-8583-88739f7b1b36","Type":"ContainerDied","Data":"434b98b8e4f180c2881a80e3c39a334a69228320065d4ba149eaec48c88f6d68"}
Sep 11 18:09:58.064423 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.064403    2559 scope.go:117] "RemoveContainer" containerID="41dce4b05b3ed1f46397572e5682ae49559b80878e75479bb566f2b13d51f514"
Sep 11 18:09:58.076249 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.076225    2559 scope.go:117] "RemoveContainer" containerID="41dce4b05b3ed1f46397572e5682ae49559b80878e75479bb566f2b13d51f514"
Sep 11 18:09:58.076642 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:09:58.076616    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41dce4b05b3ed1f46397572e5682ae49559b80878e75479bb566f2b13d51f514\": container with ID starting with 41dce4b05b3ed1f46397572e5682ae49559b80878e75479bb566f2b13d51f514 not found: ID does not exist" containerID="41dce4b05b3ed1f46397572e5682ae49559b80878e75479bb566f2b13d51f514"
Sep 11 18:09:58.076739 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.076654    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41dce4b05b3ed1f46397572e5682ae49559b80878e75479bb566f2b13d51f514"} err="failed to get container status \"41dce4b05b3ed1f46397572e5682ae49559b80878e75479bb566f2b13d51f514\": rpc error: code = NotFound desc = could not find container \"41dce4b05b3ed1f46397572e5682ae49559b80878e75479bb566f2b13d51f514\": container with ID starting with 41dce4b05b3ed1f46397572e5682ae49559b80878e75479bb566f2b13d51f514 not found: ID does not exist"
Sep 11 18:09:58.097378 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.097327    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"]
Sep 11 18:09:58.101787 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.101760    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-386935/sub-test-model-386935-kserve-995f54b75-mplbx"]
Sep 11 18:09:58.165071 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.165032    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a871d8a2-a8ed-44e8-8583-88739f7b1b36-tls-certs\") pod \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\" (UID: \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\") "
Sep 11 18:09:58.165071 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.165076    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a871d8a2-a8ed-44e8-8583-88739f7b1b36-dshm\") pod \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\" (UID: \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\") "
Sep 11 18:09:58.165289 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.165106    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a871d8a2-a8ed-44e8-8583-88739f7b1b36-home\") pod \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\" (UID: \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\") "
Sep 11 18:09:58.165289 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.165173    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a871d8a2-a8ed-44e8-8583-88739f7b1b36-tmp-dir\") pod \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\" (UID: \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\") "
Sep 11 18:09:58.165289 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.165208    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a871d8a2-a8ed-44e8-8583-88739f7b1b36-model-cache\") pod \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\" (UID: \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\") "
Sep 11 18:09:58.165289 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.165272    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7lcc\" (UniqueName: \"kubernetes.io/projected/a871d8a2-a8ed-44e8-8583-88739f7b1b36-kube-api-access-s7lcc\") pod \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\" (UID: \"a871d8a2-a8ed-44e8-8583-88739f7b1b36\") "
Sep 11 18:09:58.165514 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.165493    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a871d8a2-a8ed-44e8-8583-88739f7b1b36-home" (OuterVolumeSpecName: "home") pod "a871d8a2-a8ed-44e8-8583-88739f7b1b36" (UID: "a871d8a2-a8ed-44e8-8583-88739f7b1b36"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:09:58.165576 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.165537    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a871d8a2-a8ed-44e8-8583-88739f7b1b36-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a871d8a2-a8ed-44e8-8583-88739f7b1b36" (UID: "a871d8a2-a8ed-44e8-8583-88739f7b1b36"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:09:58.165576 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.165549    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a871d8a2-a8ed-44e8-8583-88739f7b1b36-model-cache" (OuterVolumeSpecName: "model-cache") pod "a871d8a2-a8ed-44e8-8583-88739f7b1b36" (UID: "a871d8a2-a8ed-44e8-8583-88739f7b1b36"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:09:58.165692 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.165644    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a871d8a2-a8ed-44e8-8583-88739f7b1b36-home\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:09:58.165692 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.165660    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a871d8a2-a8ed-44e8-8583-88739f7b1b36-tmp-dir\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:09:58.165692 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.165672    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a871d8a2-a8ed-44e8-8583-88739f7b1b36-model-cache\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:09:58.167556 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.167529    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a871d8a2-a8ed-44e8-8583-88739f7b1b36-kube-api-access-s7lcc" (OuterVolumeSpecName: "kube-api-access-s7lcc") pod "a871d8a2-a8ed-44e8-8583-88739f7b1b36" (UID: "a871d8a2-a8ed-44e8-8583-88739f7b1b36"). InnerVolumeSpecName "kube-api-access-s7lcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:09:58.167731 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.167705    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a871d8a2-a8ed-44e8-8583-88739f7b1b36-dshm" (OuterVolumeSpecName: "dshm") pod "a871d8a2-a8ed-44e8-8583-88739f7b1b36" (UID: "a871d8a2-a8ed-44e8-8583-88739f7b1b36"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:09:58.167731 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.167711    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a871d8a2-a8ed-44e8-8583-88739f7b1b36-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a871d8a2-a8ed-44e8-8583-88739f7b1b36" (UID: "a871d8a2-a8ed-44e8-8583-88739f7b1b36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:09:58.266485 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.266442    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7lcc\" (UniqueName: \"kubernetes.io/projected/a871d8a2-a8ed-44e8-8583-88739f7b1b36-kube-api-access-s7lcc\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:09:58.266485 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.266477    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a871d8a2-a8ed-44e8-8583-88739f7b1b36-tls-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:09:58.266485 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.266489    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a871d8a2-a8ed-44e8-8583-88739f7b1b36-dshm\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:09:58.385307 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.385271    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"]
Sep 11 18:09:58.390395 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.390342    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ebea5/sub-test-model-3ebea5-kserve-7869bb844f-fvnt7"]
Sep 11 18:09:58.786263 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.786227    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"]
Sep 11 18:09:58.786666 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.786653    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff00d624-6d9b-4e80-bd6f-54343473e6e1" containerName="main"
Sep 11 18:09:58.786720 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.786668    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff00d624-6d9b-4e80-bd6f-54343473e6e1" containerName="main"
Sep 11 18:09:58.786720 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.786683    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a871d8a2-a8ed-44e8-8583-88739f7b1b36" containerName="main"
Sep 11 18:09:58.786720 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.786689    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a871d8a2-a8ed-44e8-8583-88739f7b1b36" containerName="main"
Sep 11 18:09:58.786820 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.786749    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a871d8a2-a8ed-44e8-8583-88739f7b1b36" containerName="main"
Sep 11 18:09:58.786820 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.786760    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff00d624-6d9b-4e80-bd6f-54343473e6e1" containerName="main"
Sep 11 18:09:58.791034 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.791012    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:09:58.793940 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.793914    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-386935\"/\"rate-test-model-386935-kserve-self-signed-certs\""
Sep 11 18:09:58.794580 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.794562    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-386935\"/\"kube-root-ca.crt\""
Sep 11 18:09:58.794650 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.794562    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-386935\"/\"default-dockercfg-2787z\""
Sep 11 18:09:58.794650 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.794597    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-386935\"/\"openshift-service-ca.crt\""
Sep 11 18:09:58.813240 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.813208    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"]
Sep 11 18:09:58.871724 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.871691    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f7a2e9-07ef-4da5-835d-0ede48937e49-tmp-dir\") pod \"rate-test-model-386935-kserve-6fdccf99c9-f765m\" (UID: \"27f7a2e9-07ef-4da5-835d-0ede48937e49\") " pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:09:58.871975 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.871742    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f7a2e9-07ef-4da5-835d-0ede48937e49-dshm\") pod \"rate-test-model-386935-kserve-6fdccf99c9-f765m\" (UID: \"27f7a2e9-07ef-4da5-835d-0ede48937e49\") " pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:09:58.871975 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.871821    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmvtv\" (UniqueName: \"kubernetes.io/projected/27f7a2e9-07ef-4da5-835d-0ede48937e49-kube-api-access-jmvtv\") pod \"rate-test-model-386935-kserve-6fdccf99c9-f765m\" (UID: \"27f7a2e9-07ef-4da5-835d-0ede48937e49\") " pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:09:58.871975 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.871864    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f7a2e9-07ef-4da5-835d-0ede48937e49-model-cache\") pod \"rate-test-model-386935-kserve-6fdccf99c9-f765m\" (UID: \"27f7a2e9-07ef-4da5-835d-0ede48937e49\") " pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:09:58.871975 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.871895    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f7a2e9-07ef-4da5-835d-0ede48937e49-home\") pod \"rate-test-model-386935-kserve-6fdccf99c9-f765m\" (UID: \"27f7a2e9-07ef-4da5-835d-0ede48937e49\") " pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:09:58.872169 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.871980    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f7a2e9-07ef-4da5-835d-0ede48937e49-tls-certs\") pod \"rate-test-model-386935-kserve-6fdccf99c9-f765m\" (UID: \"27f7a2e9-07ef-4da5-835d-0ede48937e49\") " pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:09:58.973023 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.972973    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f7a2e9-07ef-4da5-835d-0ede48937e49-dshm\") pod \"rate-test-model-386935-kserve-6fdccf99c9-f765m\" (UID: \"27f7a2e9-07ef-4da5-835d-0ede48937e49\") " pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:09:58.973180 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.973129    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmvtv\" (UniqueName: \"kubernetes.io/projected/27f7a2e9-07ef-4da5-835d-0ede48937e49-kube-api-access-jmvtv\") pod \"rate-test-model-386935-kserve-6fdccf99c9-f765m\" (UID: \"27f7a2e9-07ef-4da5-835d-0ede48937e49\") " pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:09:58.973180 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.973161    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f7a2e9-07ef-4da5-835d-0ede48937e49-model-cache\") pod \"rate-test-model-386935-kserve-6fdccf99c9-f765m\" (UID: \"27f7a2e9-07ef-4da5-835d-0ede48937e49\") " pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:09:58.973318 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.973201    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f7a2e9-07ef-4da5-835d-0ede48937e49-home\") pod \"rate-test-model-386935-kserve-6fdccf99c9-f765m\" (UID: \"27f7a2e9-07ef-4da5-835d-0ede48937e49\") " pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:09:58.973318 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.973257    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f7a2e9-07ef-4da5-835d-0ede48937e49-tls-certs\") pod \"rate-test-model-386935-kserve-6fdccf99c9-f765m\" (UID: \"27f7a2e9-07ef-4da5-835d-0ede48937e49\") " pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:09:58.973318 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.973296    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f7a2e9-07ef-4da5-835d-0ede48937e49-tmp-dir\") pod \"rate-test-model-386935-kserve-6fdccf99c9-f765m\" (UID: \"27f7a2e9-07ef-4da5-835d-0ede48937e49\") " pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:09:58.973640 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.973616    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f7a2e9-07ef-4da5-835d-0ede48937e49-model-cache\") pod \"rate-test-model-386935-kserve-6fdccf99c9-f765m\" (UID: \"27f7a2e9-07ef-4da5-835d-0ede48937e49\") " pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:09:58.973710 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.973640    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f7a2e9-07ef-4da5-835d-0ede48937e49-home\") pod \"rate-test-model-386935-kserve-6fdccf99c9-f765m\" (UID: \"27f7a2e9-07ef-4da5-835d-0ede48937e49\") " pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:09:58.973710 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.973692    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f7a2e9-07ef-4da5-835d-0ede48937e49-tmp-dir\") pod \"rate-test-model-386935-kserve-6fdccf99c9-f765m\" (UID: \"27f7a2e9-07ef-4da5-835d-0ede48937e49\") " pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:09:58.975301 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.975278    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f7a2e9-07ef-4da5-835d-0ede48937e49-dshm\") pod \"rate-test-model-386935-kserve-6fdccf99c9-f765m\" (UID: \"27f7a2e9-07ef-4da5-835d-0ede48937e49\") " pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:09:58.975733 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.975715    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f7a2e9-07ef-4da5-835d-0ede48937e49-tls-certs\") pod \"rate-test-model-386935-kserve-6fdccf99c9-f765m\" (UID: \"27f7a2e9-07ef-4da5-835d-0ede48937e49\") " pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:09:58.982747 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:58.982724    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmvtv\" (UniqueName: \"kubernetes.io/projected/27f7a2e9-07ef-4da5-835d-0ede48937e49-kube-api-access-jmvtv\") pod \"rate-test-model-386935-kserve-6fdccf99c9-f765m\" (UID: \"27f7a2e9-07ef-4da5-835d-0ede48937e49\") " pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:09:59.053883 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:59.053803    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a871d8a2-a8ed-44e8-8583-88739f7b1b36" path="/var/lib/kubelet/pods/a871d8a2-a8ed-44e8-8583-88739f7b1b36/volumes"
Sep 11 18:09:59.054224 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:59.054209    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff00d624-6d9b-4e80-bd6f-54343473e6e1" path="/var/lib/kubelet/pods/ff00d624-6d9b-4e80-bd6f-54343473e6e1/volumes"
Sep 11 18:09:59.100803 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:59.100769    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:09:59.389764 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:09:59.389732    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"]
Sep 11 18:09:59.398894 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:09:59.398864    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27f7a2e9_07ef_4da5_835d_0ede48937e49.slice/crio-a810124cce75c660c1fefa41ff4ef9ee2ae82765385c85f3b91e0d9cfbe32a00 WatchSource:0}: Error finding container a810124cce75c660c1fefa41ff4ef9ee2ae82765385c85f3b91e0d9cfbe32a00: Status 404 returned error can't find the container with id a810124cce75c660c1fefa41ff4ef9ee2ae82765385c85f3b91e0d9cfbe32a00
Sep 11 18:10:00.070238 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:00.070201    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m" event={"ID":"27f7a2e9-07ef-4da5-835d-0ede48937e49","Type":"ContainerStarted","Data":"d24cdcfbb8f39b6f0a6a1d9af947e95defcc02ce5cb6a85a1661bf4cd27af1bc"}
Sep 11 18:10:00.070238 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:00.070246    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m" event={"ID":"27f7a2e9-07ef-4da5-835d-0ede48937e49","Type":"ContainerStarted","Data":"a810124cce75c660c1fefa41ff4ef9ee2ae82765385c85f3b91e0d9cfbe32a00"}
Sep 11 18:10:00.099640 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:00.099580    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m" podStartSLOduration=2.099564015 podStartE2EDuration="2.099564015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:10:00.097011419 +0000 UTC m=+1505.542111989" watchObservedRunningTime="2026-09-11 18:10:00.099564015 +0000 UTC m=+1505.544664586"
Sep 11 18:10:04.666661 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:04.666623    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6"]
Sep 11 18:10:04.667066 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:04.666868    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6" podUID="ad247e53-5db0-4caa-8865-e4696eb756ff" containerName="maas-api" containerID="cri-o://39c1508f463d5af3e5eeb2b1e65d0caff5ec0c2735287f8a0e3fe266dbe5ab2c" gracePeriod=30
Sep 11 18:10:04.729122 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:04.729082    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5"]
Sep 11 18:10:04.729393 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:04.729342    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5" podUID="0d7a8e0f-f34c-4fb2-aae2-c8d16b118916" containerName="payload-processing" containerID="cri-o://7139259763b46d9867bb2f7f402ba9c6d0279a8ac74c81a845eb2d1c103fa665" gracePeriod=30
Sep 11 18:10:04.735871 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:04.735847    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw"]
Sep 11 18:10:04.736093 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:04.736069    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw" podUID="da6ccef1-d0e9-4e74-9cd1-cf7ac841c151" containerName="payload-pre-processing" containerID="cri-o://734351ed5870b1b28f4395250f36cb41dcc29529808155ae554597396ca7d159" gracePeriod=30
Sep 11 18:10:04.985190 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:04.985159    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6"
Sep 11 18:10:05.021663 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.021626    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad247e53-5db0-4caa-8865-e4696eb756ff-maas-api-tls\") pod \"ad247e53-5db0-4caa-8865-e4696eb756ff\" (UID: \"ad247e53-5db0-4caa-8865-e4696eb756ff\") "
Sep 11 18:10:05.021862 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.021680    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad247e53-5db0-4caa-8865-e4696eb756ff-openshift-service-ca\") pod \"ad247e53-5db0-4caa-8865-e4696eb756ff\" (UID: \"ad247e53-5db0-4caa-8865-e4696eb756ff\") "
Sep 11 18:10:05.021862 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.021732    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad247e53-5db0-4caa-8865-e4696eb756ff-metrics-certs\") pod \"ad247e53-5db0-4caa-8865-e4696eb756ff\" (UID: \"ad247e53-5db0-4caa-8865-e4696eb756ff\") "
Sep 11 18:10:05.021862 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.021793    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phmgp\" (UniqueName: \"kubernetes.io/projected/ad247e53-5db0-4caa-8865-e4696eb756ff-kube-api-access-phmgp\") pod \"ad247e53-5db0-4caa-8865-e4696eb756ff\" (UID: \"ad247e53-5db0-4caa-8865-e4696eb756ff\") "
Sep 11 18:10:05.022195 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.022162    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad247e53-5db0-4caa-8865-e4696eb756ff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ad247e53-5db0-4caa-8865-e4696eb756ff" (UID: "ad247e53-5db0-4caa-8865-e4696eb756ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:10:05.024047 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.024016    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad247e53-5db0-4caa-8865-e4696eb756ff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ad247e53-5db0-4caa-8865-e4696eb756ff" (UID: "ad247e53-5db0-4caa-8865-e4696eb756ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:10:05.024185 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.024046    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad247e53-5db0-4caa-8865-e4696eb756ff-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ad247e53-5db0-4caa-8865-e4696eb756ff" (UID: "ad247e53-5db0-4caa-8865-e4696eb756ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:10:05.024185 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.024067    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad247e53-5db0-4caa-8865-e4696eb756ff-kube-api-access-phmgp" (OuterVolumeSpecName: "kube-api-access-phmgp") pod "ad247e53-5db0-4caa-8865-e4696eb756ff" (UID: "ad247e53-5db0-4caa-8865-e4696eb756ff"). InnerVolumeSpecName "kube-api-access-phmgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:10:05.036133 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.036106    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5"
Sep 11 18:10:05.038618 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.038595    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw"
Sep 11 18:10:05.100480 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.100451    2559 generic.go:356] "Generic (PLEG): container finished" podID="da6ccef1-d0e9-4e74-9cd1-cf7ac841c151" containerID="734351ed5870b1b28f4395250f36cb41dcc29529808155ae554597396ca7d159" exitCode=0
Sep 11 18:10:05.100659 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.100498    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw"
Sep 11 18:10:05.100659 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.100545    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw" event={"ID":"da6ccef1-d0e9-4e74-9cd1-cf7ac841c151","Type":"ContainerDied","Data":"734351ed5870b1b28f4395250f36cb41dcc29529808155ae554597396ca7d159"}
Sep 11 18:10:05.100659 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.100599    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw" event={"ID":"da6ccef1-d0e9-4e74-9cd1-cf7ac841c151","Type":"ContainerDied","Data":"1f6d7b3f509cd0e0af7f38789381b70e6a5032072742b6775136f83fe051f934"}
Sep 11 18:10:05.100659 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.100626    2559 scope.go:117] "RemoveContainer" containerID="734351ed5870b1b28f4395250f36cb41dcc29529808155ae554597396ca7d159"
Sep 11 18:10:05.102101 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.102077    2559 generic.go:356] "Generic (PLEG): container finished" podID="ad247e53-5db0-4caa-8865-e4696eb756ff" containerID="39c1508f463d5af3e5eeb2b1e65d0caff5ec0c2735287f8a0e3fe266dbe5ab2c" exitCode=0
Sep 11 18:10:05.102238 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.102106    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6" event={"ID":"ad247e53-5db0-4caa-8865-e4696eb756ff","Type":"ContainerDied","Data":"39c1508f463d5af3e5eeb2b1e65d0caff5ec0c2735287f8a0e3fe266dbe5ab2c"}
Sep 11 18:10:05.102238 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.102126    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6"
Sep 11 18:10:05.102238 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.102148    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6" event={"ID":"ad247e53-5db0-4caa-8865-e4696eb756ff","Type":"ContainerDied","Data":"944f50eb6e2d342bd91902e999e92cb7d902c5dc6bc9ff3436e318da1a5e630d"}
Sep 11 18:10:05.103834 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.103811    2559 generic.go:356] "Generic (PLEG): container finished" podID="0d7a8e0f-f34c-4fb2-aae2-c8d16b118916" containerID="7139259763b46d9867bb2f7f402ba9c6d0279a8ac74c81a845eb2d1c103fa665" exitCode=0
Sep 11 18:10:05.103938 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.103856    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5"
Sep 11 18:10:05.103995 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.103855    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5" event={"ID":"0d7a8e0f-f34c-4fb2-aae2-c8d16b118916","Type":"ContainerDied","Data":"7139259763b46d9867bb2f7f402ba9c6d0279a8ac74c81a845eb2d1c103fa665"}
Sep 11 18:10:05.104043 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.104004    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5" event={"ID":"0d7a8e0f-f34c-4fb2-aae2-c8d16b118916","Type":"ContainerDied","Data":"3b56d945d4ed2c47bc4421ece5d76fdd1e0b8ef5f94e1b3d2951c8ada32dcebc"}
Sep 11 18:10:05.111762 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.111742    2559 scope.go:117] "RemoveContainer" containerID="734351ed5870b1b28f4395250f36cb41dcc29529808155ae554597396ca7d159"
Sep 11 18:10:05.112188 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:10:05.112158    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"734351ed5870b1b28f4395250f36cb41dcc29529808155ae554597396ca7d159\": container with ID starting with 734351ed5870b1b28f4395250f36cb41dcc29529808155ae554597396ca7d159 not found: ID does not exist" containerID="734351ed5870b1b28f4395250f36cb41dcc29529808155ae554597396ca7d159"
Sep 11 18:10:05.112452 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.112200    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"734351ed5870b1b28f4395250f36cb41dcc29529808155ae554597396ca7d159"} err="failed to get container status \"734351ed5870b1b28f4395250f36cb41dcc29529808155ae554597396ca7d159\": rpc error: code = NotFound desc = could not find container \"734351ed5870b1b28f4395250f36cb41dcc29529808155ae554597396ca7d159\": container with ID starting with 734351ed5870b1b28f4395250f36cb41dcc29529808155ae554597396ca7d159 not found: ID does not exist"
Sep 11 18:10:05.112452 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.112222    2559 scope.go:117] "RemoveContainer" containerID="39c1508f463d5af3e5eeb2b1e65d0caff5ec0c2735287f8a0e3fe266dbe5ab2c"
Sep 11 18:10:05.125043 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.124830    2559 scope.go:117] "RemoveContainer" containerID="39c1508f463d5af3e5eeb2b1e65d0caff5ec0c2735287f8a0e3fe266dbe5ab2c"
Sep 11 18:10:05.125043 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.125029    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5658r\" (UniqueName: \"kubernetes.io/projected/0d7a8e0f-f34c-4fb2-aae2-c8d16b118916-kube-api-access-5658r\") pod \"0d7a8e0f-f34c-4fb2-aae2-c8d16b118916\" (UID: \"0d7a8e0f-f34c-4fb2-aae2-c8d16b118916\") "
Sep 11 18:10:05.125489 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.125074    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7qc2\" (UniqueName: \"kubernetes.io/projected/da6ccef1-d0e9-4e74-9cd1-cf7ac841c151-kube-api-access-n7qc2\") pod \"da6ccef1-d0e9-4e74-9cd1-cf7ac841c151\" (UID: \"da6ccef1-d0e9-4e74-9cd1-cf7ac841c151\") "
Sep 11 18:10:05.125489 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:10:05.125396    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c1508f463d5af3e5eeb2b1e65d0caff5ec0c2735287f8a0e3fe266dbe5ab2c\": container with ID starting with 39c1508f463d5af3e5eeb2b1e65d0caff5ec0c2735287f8a0e3fe266dbe5ab2c not found: ID does not exist" containerID="39c1508f463d5af3e5eeb2b1e65d0caff5ec0c2735287f8a0e3fe266dbe5ab2c"
Sep 11 18:10:05.125489 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.125424    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phmgp\" (UniqueName: \"kubernetes.io/projected/ad247e53-5db0-4caa-8865-e4696eb756ff-kube-api-access-phmgp\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:10:05.125489 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.125455    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad247e53-5db0-4caa-8865-e4696eb756ff-maas-api-tls\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:10:05.125489 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.125469    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad247e53-5db0-4caa-8865-e4696eb756ff-openshift-service-ca\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:10:05.125489 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.125481    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad247e53-5db0-4caa-8865-e4696eb756ff-metrics-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:10:05.125489 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.125426    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c1508f463d5af3e5eeb2b1e65d0caff5ec0c2735287f8a0e3fe266dbe5ab2c"} err="failed to get container status \"39c1508f463d5af3e5eeb2b1e65d0caff5ec0c2735287f8a0e3fe266dbe5ab2c\": rpc error: code = NotFound desc = could not find container \"39c1508f463d5af3e5eeb2b1e65d0caff5ec0c2735287f8a0e3fe266dbe5ab2c\": container with ID starting with 39c1508f463d5af3e5eeb2b1e65d0caff5ec0c2735287f8a0e3fe266dbe5ab2c not found: ID does not exist"
Sep 11 18:10:05.125840 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.125505    2559 scope.go:117] "RemoveContainer" containerID="7139259763b46d9867bb2f7f402ba9c6d0279a8ac74c81a845eb2d1c103fa665"
Sep 11 18:10:05.127611 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.127589    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d7a8e0f-f34c-4fb2-aae2-c8d16b118916-kube-api-access-5658r" (OuterVolumeSpecName: "kube-api-access-5658r") pod "0d7a8e0f-f34c-4fb2-aae2-c8d16b118916" (UID: "0d7a8e0f-f34c-4fb2-aae2-c8d16b118916"). InnerVolumeSpecName "kube-api-access-5658r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:10:05.127685 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.127663    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6ccef1-d0e9-4e74-9cd1-cf7ac841c151-kube-api-access-n7qc2" (OuterVolumeSpecName: "kube-api-access-n7qc2") pod "da6ccef1-d0e9-4e74-9cd1-cf7ac841c151" (UID: "da6ccef1-d0e9-4e74-9cd1-cf7ac841c151"). InnerVolumeSpecName "kube-api-access-n7qc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:10:05.130861 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.130836    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6"]
Sep 11 18:10:05.135407 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.135388    2559 scope.go:117] "RemoveContainer" containerID="7139259763b46d9867bb2f7f402ba9c6d0279a8ac74c81a845eb2d1c103fa665"
Sep 11 18:10:05.135741 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:10:05.135713    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7139259763b46d9867bb2f7f402ba9c6d0279a8ac74c81a845eb2d1c103fa665\": container with ID starting with 7139259763b46d9867bb2f7f402ba9c6d0279a8ac74c81a845eb2d1c103fa665 not found: ID does not exist" containerID="7139259763b46d9867bb2f7f402ba9c6d0279a8ac74c81a845eb2d1c103fa665"
Sep 11 18:10:05.135802 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.135746    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7139259763b46d9867bb2f7f402ba9c6d0279a8ac74c81a845eb2d1c103fa665"} err="failed to get container status \"7139259763b46d9867bb2f7f402ba9c6d0279a8ac74c81a845eb2d1c103fa665\": rpc error: code = NotFound desc = could not find container \"7139259763b46d9867bb2f7f402ba9c6d0279a8ac74c81a845eb2d1c103fa665\": container with ID starting with 7139259763b46d9867bb2f7f402ba9c6d0279a8ac74c81a845eb2d1c103fa665 not found: ID does not exist"
Sep 11 18:10:05.145422 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.145396    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6de04f-5b6ddbbccd-7n5f6"]
Sep 11 18:10:05.226742 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.226640    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6ccef1-d0e9-4e74-9cd1-cf7ac841c151-plugins-config-volume\") pod \"da6ccef1-d0e9-4e74-9cd1-cf7ac841c151\" (UID: \"da6ccef1-d0e9-4e74-9cd1-cf7ac841c151\") "
Sep 11 18:10:05.226742 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.226703    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d7a8e0f-f34c-4fb2-aae2-c8d16b118916-plugins-config-volume\") pod \"0d7a8e0f-f34c-4fb2-aae2-c8d16b118916\" (UID: \"0d7a8e0f-f34c-4fb2-aae2-c8d16b118916\") "
Sep 11 18:10:05.227050 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.227029    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5658r\" (UniqueName: \"kubernetes.io/projected/0d7a8e0f-f34c-4fb2-aae2-c8d16b118916-kube-api-access-5658r\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:10:05.227113 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.227056    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7qc2\" (UniqueName: \"kubernetes.io/projected/da6ccef1-d0e9-4e74-9cd1-cf7ac841c151-kube-api-access-n7qc2\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:10:05.227163 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.227108    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d7a8e0f-f34c-4fb2-aae2-c8d16b118916-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0d7a8e0f-f34c-4fb2-aae2-c8d16b118916" (UID: "0d7a8e0f-f34c-4fb2-aae2-c8d16b118916"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:10:05.227163 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.227107    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6ccef1-d0e9-4e74-9cd1-cf7ac841c151-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "da6ccef1-d0e9-4e74-9cd1-cf7ac841c151" (UID: "da6ccef1-d0e9-4e74-9cd1-cf7ac841c151"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:10:05.327648 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.327607    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6ccef1-d0e9-4e74-9cd1-cf7ac841c151-plugins-config-volume\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:10:05.327648 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.327638    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d7a8e0f-f34c-4fb2-aae2-c8d16b118916-plugins-config-volume\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:10:05.430414 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.430384    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw"]
Sep 11 18:10:05.439386 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.439333    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6de04f-7dc64cc978-2w6fw"]
Sep 11 18:10:05.462472 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.462443    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5"]
Sep 11 18:10:05.465297 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:05.465275    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6de04f-85cd47cc7c-jz4c5"]
Sep 11 18:10:06.466206 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:06.466167    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:10:07.054743 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:07.054705    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d7a8e0f-f34c-4fb2-aae2-c8d16b118916" path="/var/lib/kubelet/pods/0d7a8e0f-f34c-4fb2-aae2-c8d16b118916/volumes"
Sep 11 18:10:07.055375 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:07.055331    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad247e53-5db0-4caa-8865-e4696eb756ff" path="/var/lib/kubelet/pods/ad247e53-5db0-4caa-8865-e4696eb756ff/volumes"
Sep 11 18:10:07.055967 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:07.055945    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da6ccef1-d0e9-4e74-9cd1-cf7ac841c151" path="/var/lib/kubelet/pods/da6ccef1-d0e9-4e74-9cd1-cf7ac841c151/volumes"
Sep 11 18:10:09.101430 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:09.101390    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:10:09.101931 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:09.101670    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:10:09.114550 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:09.114518    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:10:09.143832 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:09.143802    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:10:19.999153 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:19.999071    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:10:22.517612 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.517577    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"]
Sep 11 18:10:22.518157 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.518136    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da6ccef1-d0e9-4e74-9cd1-cf7ac841c151" containerName="payload-pre-processing"
Sep 11 18:10:22.518242 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.518161    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="da6ccef1-d0e9-4e74-9cd1-cf7ac841c151" containerName="payload-pre-processing"
Sep 11 18:10:22.518242 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.518175    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad247e53-5db0-4caa-8865-e4696eb756ff" containerName="maas-api"
Sep 11 18:10:22.518242 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.518184    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad247e53-5db0-4caa-8865-e4696eb756ff" containerName="maas-api"
Sep 11 18:10:22.518242 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.518196    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d7a8e0f-f34c-4fb2-aae2-c8d16b118916" containerName="payload-processing"
Sep 11 18:10:22.518242 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.518205    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d7a8e0f-f34c-4fb2-aae2-c8d16b118916" containerName="payload-processing"
Sep 11 18:10:22.518542 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.518319    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d7a8e0f-f34c-4fb2-aae2-c8d16b118916" containerName="payload-processing"
Sep 11 18:10:22.518542 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.518335    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad247e53-5db0-4caa-8865-e4696eb756ff" containerName="maas-api"
Sep 11 18:10:22.518542 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.518363    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="da6ccef1-d0e9-4e74-9cd1-cf7ac841c151" containerName="payload-pre-processing"
Sep 11 18:10:22.522899 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.522878    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:10:22.528342 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.528309    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3ebea5\"/\"kube-root-ca.crt\""
Sep 11 18:10:22.529427 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.529253    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3ebea5\"/\"openshift-service-ca.crt\""
Sep 11 18:10:22.529427 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.529309    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3ebea5\"/\"rate-test-model-3ebea5-kserve-self-signed-certs\""
Sep 11 18:10:22.529427 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.529344    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3ebea5\"/\"default-dockercfg-5wbch\""
Sep 11 18:10:22.566679 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.566633    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"]
Sep 11 18:10:22.574474 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.574444    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4bbb691-9523-4c4b-810b-c9dec3424c07-tmp-dir\") pod \"rate-test-model-3ebea5-kserve-9d4bc-wlp82\" (UID: \"d4bbb691-9523-4c4b-810b-c9dec3424c07\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:10:22.574634 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.574488    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4bbb691-9523-4c4b-810b-c9dec3424c07-tls-certs\") pod \"rate-test-model-3ebea5-kserve-9d4bc-wlp82\" (UID: \"d4bbb691-9523-4c4b-810b-c9dec3424c07\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:10:22.574634 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.574520    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4bbb691-9523-4c4b-810b-c9dec3424c07-model-cache\") pod \"rate-test-model-3ebea5-kserve-9d4bc-wlp82\" (UID: \"d4bbb691-9523-4c4b-810b-c9dec3424c07\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:10:22.574634 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.574570    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4bbb691-9523-4c4b-810b-c9dec3424c07-dshm\") pod \"rate-test-model-3ebea5-kserve-9d4bc-wlp82\" (UID: \"d4bbb691-9523-4c4b-810b-c9dec3424c07\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:10:22.574634 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.574587    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4bbb691-9523-4c4b-810b-c9dec3424c07-home\") pod \"rate-test-model-3ebea5-kserve-9d4bc-wlp82\" (UID: \"d4bbb691-9523-4c4b-810b-c9dec3424c07\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:10:22.574888 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.574643    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6q6d\" (UniqueName: \"kubernetes.io/projected/d4bbb691-9523-4c4b-810b-c9dec3424c07-kube-api-access-d6q6d\") pod \"rate-test-model-3ebea5-kserve-9d4bc-wlp82\" (UID: \"d4bbb691-9523-4c4b-810b-c9dec3424c07\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:10:22.675393 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.675338    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4bbb691-9523-4c4b-810b-c9dec3424c07-model-cache\") pod \"rate-test-model-3ebea5-kserve-9d4bc-wlp82\" (UID: \"d4bbb691-9523-4c4b-810b-c9dec3424c07\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:10:22.675585 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.675432    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4bbb691-9523-4c4b-810b-c9dec3424c07-dshm\") pod \"rate-test-model-3ebea5-kserve-9d4bc-wlp82\" (UID: \"d4bbb691-9523-4c4b-810b-c9dec3424c07\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:10:22.675585 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.675459    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4bbb691-9523-4c4b-810b-c9dec3424c07-home\") pod \"rate-test-model-3ebea5-kserve-9d4bc-wlp82\" (UID: \"d4bbb691-9523-4c4b-810b-c9dec3424c07\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:10:22.675585 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.675502    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6q6d\" (UniqueName: \"kubernetes.io/projected/d4bbb691-9523-4c4b-810b-c9dec3424c07-kube-api-access-d6q6d\") pod \"rate-test-model-3ebea5-kserve-9d4bc-wlp82\" (UID: \"d4bbb691-9523-4c4b-810b-c9dec3424c07\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:10:22.675745 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.675584    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4bbb691-9523-4c4b-810b-c9dec3424c07-tmp-dir\") pod \"rate-test-model-3ebea5-kserve-9d4bc-wlp82\" (UID: \"d4bbb691-9523-4c4b-810b-c9dec3424c07\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:10:22.675745 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.675628    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4bbb691-9523-4c4b-810b-c9dec3424c07-tls-certs\") pod \"rate-test-model-3ebea5-kserve-9d4bc-wlp82\" (UID: \"d4bbb691-9523-4c4b-810b-c9dec3424c07\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:10:22.675930 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.675904    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4bbb691-9523-4c4b-810b-c9dec3424c07-model-cache\") pod \"rate-test-model-3ebea5-kserve-9d4bc-wlp82\" (UID: \"d4bbb691-9523-4c4b-810b-c9dec3424c07\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:10:22.675992 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.675932    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4bbb691-9523-4c4b-810b-c9dec3424c07-home\") pod \"rate-test-model-3ebea5-kserve-9d4bc-wlp82\" (UID: \"d4bbb691-9523-4c4b-810b-c9dec3424c07\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:10:22.675992 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.675943    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4bbb691-9523-4c4b-810b-c9dec3424c07-tmp-dir\") pod \"rate-test-model-3ebea5-kserve-9d4bc-wlp82\" (UID: \"d4bbb691-9523-4c4b-810b-c9dec3424c07\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:10:22.677775 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.677749    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4bbb691-9523-4c4b-810b-c9dec3424c07-dshm\") pod \"rate-test-model-3ebea5-kserve-9d4bc-wlp82\" (UID: \"d4bbb691-9523-4c4b-810b-c9dec3424c07\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:10:22.678031 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.678014    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4bbb691-9523-4c4b-810b-c9dec3424c07-tls-certs\") pod \"rate-test-model-3ebea5-kserve-9d4bc-wlp82\" (UID: \"d4bbb691-9523-4c4b-810b-c9dec3424c07\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:10:22.722811 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.722773    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6q6d\" (UniqueName: \"kubernetes.io/projected/d4bbb691-9523-4c4b-810b-c9dec3424c07-kube-api-access-d6q6d\") pod \"rate-test-model-3ebea5-kserve-9d4bc-wlp82\" (UID: \"d4bbb691-9523-4c4b-810b-c9dec3424c07\") " pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:10:22.832191 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:22.832102    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:10:23.130120 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:23.130090    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"]
Sep 11 18:10:23.131160 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:10:23.131133    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4bbb691_9523_4c4b_810b_c9dec3424c07.slice/crio-c2a29378ace99331bace805f3ea8437069f047add6dc7ed6ce1160382ec0337f WatchSource:0}: Error finding container c2a29378ace99331bace805f3ea8437069f047add6dc7ed6ce1160382ec0337f: Status 404 returned error can't find the container with id c2a29378ace99331bace805f3ea8437069f047add6dc7ed6ce1160382ec0337f
Sep 11 18:10:23.211361 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:23.211309    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82" event={"ID":"d4bbb691-9523-4c4b-810b-c9dec3424c07","Type":"ContainerStarted","Data":"b0a83488df15ae541bf350a037fefa3bb317b413d11d4d90f0d868be06964e72"}
Sep 11 18:10:23.211508 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:23.211371    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82" event={"ID":"d4bbb691-9523-4c4b-810b-c9dec3424c07","Type":"ContainerStarted","Data":"c2a29378ace99331bace805f3ea8437069f047add6dc7ed6ce1160382ec0337f"}
Sep 11 18:10:29.201651 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:29.201593    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82" podStartSLOduration=7.201574118 podStartE2EDuration="7.201574118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:10:23.249813329 +0000 UTC m=+1528.694913899" watchObservedRunningTime="2026-09-11 18:10:29.201574118 +0000 UTC m=+1534.646674688"
Sep 11 18:10:29.202270 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:29.202252    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v"]
Sep 11 18:10:29.202486 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:29.202464    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v" podUID="472fa464-1370-4d6d-a851-dd1159a7708d" containerName="payload-processing" containerID="cri-o://b1f6b9490f63bd582f784b7a9fbf43e44766f8db411691cff383f67c226dcd89" gracePeriod=30
Sep 11 18:10:29.463563 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:29.463498    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v"
Sep 11 18:10:29.544608 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:29.544566    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472fa464-1370-4d6d-a851-dd1159a7708d-plugins-config-volume\") pod \"472fa464-1370-4d6d-a851-dd1159a7708d\" (UID: \"472fa464-1370-4d6d-a851-dd1159a7708d\") "
Sep 11 18:10:29.544786 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:29.544761    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqdnf\" (UniqueName: \"kubernetes.io/projected/472fa464-1370-4d6d-a851-dd1159a7708d-kube-api-access-qqdnf\") pod \"472fa464-1370-4d6d-a851-dd1159a7708d\" (UID: \"472fa464-1370-4d6d-a851-dd1159a7708d\") "
Sep 11 18:10:29.545041 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:29.545014    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/472fa464-1370-4d6d-a851-dd1159a7708d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "472fa464-1370-4d6d-a851-dd1159a7708d" (UID: "472fa464-1370-4d6d-a851-dd1159a7708d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:10:29.546847 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:29.546823    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472fa464-1370-4d6d-a851-dd1159a7708d-kube-api-access-qqdnf" (OuterVolumeSpecName: "kube-api-access-qqdnf") pod "472fa464-1370-4d6d-a851-dd1159a7708d" (UID: "472fa464-1370-4d6d-a851-dd1159a7708d"). InnerVolumeSpecName "kube-api-access-qqdnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:10:29.645614 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:29.645568    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472fa464-1370-4d6d-a851-dd1159a7708d-plugins-config-volume\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:10:29.645614 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:29.645605    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqdnf\" (UniqueName: \"kubernetes.io/projected/472fa464-1370-4d6d-a851-dd1159a7708d-kube-api-access-qqdnf\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:10:30.252613 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:30.252570    2559 generic.go:356] "Generic (PLEG): container finished" podID="472fa464-1370-4d6d-a851-dd1159a7708d" containerID="b1f6b9490f63bd582f784b7a9fbf43e44766f8db411691cff383f67c226dcd89" exitCode=0
Sep 11 18:10:30.255141 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:30.252620    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v"
Sep 11 18:10:30.255141 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:30.252651    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v" event={"ID":"472fa464-1370-4d6d-a851-dd1159a7708d","Type":"ContainerDied","Data":"b1f6b9490f63bd582f784b7a9fbf43e44766f8db411691cff383f67c226dcd89"}
Sep 11 18:10:30.255141 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:30.252705    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v" event={"ID":"472fa464-1370-4d6d-a851-dd1159a7708d","Type":"ContainerDied","Data":"b2553b285cccd365466cebd6becaa7f19aeafaf7ba8360daea2e04462006b7c5"}
Sep 11 18:10:30.255141 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:30.252727    2559 scope.go:117] "RemoveContainer" containerID="b1f6b9490f63bd582f784b7a9fbf43e44766f8db411691cff383f67c226dcd89"
Sep 11 18:10:30.262178 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:30.262159    2559 scope.go:117] "RemoveContainer" containerID="b1f6b9490f63bd582f784b7a9fbf43e44766f8db411691cff383f67c226dcd89"
Sep 11 18:10:30.262462 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:10:30.262443    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f6b9490f63bd582f784b7a9fbf43e44766f8db411691cff383f67c226dcd89\": container with ID starting with b1f6b9490f63bd582f784b7a9fbf43e44766f8db411691cff383f67c226dcd89 not found: ID does not exist" containerID="b1f6b9490f63bd582f784b7a9fbf43e44766f8db411691cff383f67c226dcd89"
Sep 11 18:10:30.262541 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:30.262470    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f6b9490f63bd582f784b7a9fbf43e44766f8db411691cff383f67c226dcd89"} err="failed to get container status \"b1f6b9490f63bd582f784b7a9fbf43e44766f8db411691cff383f67c226dcd89\": rpc error: code = NotFound desc = could not find container \"b1f6b9490f63bd582f784b7a9fbf43e44766f8db411691cff383f67c226dcd89\": container with ID starting with b1f6b9490f63bd582f784b7a9fbf43e44766f8db411691cff383f67c226dcd89 not found: ID does not exist"
Sep 11 18:10:30.290651 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:30.290623    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v"]
Sep 11 18:10:30.301776 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:30.301750    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f1705a-7d87f8f57c-4b97v"]
Sep 11 18:10:31.053276 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:31.053242    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="472fa464-1370-4d6d-a851-dd1159a7708d" path="/var/lib/kubelet/pods/472fa464-1370-4d6d-a851-dd1159a7708d/volumes"
Sep 11 18:10:32.832723 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:32.832681    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:10:32.832723 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:32.832732    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:10:32.845240 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:32.845214    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:10:33.284136 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:33.284098    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:10:34.945218 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:34.945177    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:10:50.247902 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:10:50.247862    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:11:05.234108 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:05.234066    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:11:07.600956 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:07.600923    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz"]
Sep 11 18:11:07.601446 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:07.601158    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz" podUID="1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09" containerName="maas-api" containerID="cri-o://5f4d53ff4699c833ec6318ac3fcf68b4ae9afb3ac77203998ff28f2ab01b22c8" gracePeriod=30
Sep 11 18:11:07.857808 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:07.857745    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz"
Sep 11 18:11:07.920967 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:07.920922    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09-maas-api-tls\") pod \"1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09\" (UID: \"1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09\") "
Sep 11 18:11:07.921174 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:07.920979    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5jdq\" (UniqueName: \"kubernetes.io/projected/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09-kube-api-access-c5jdq\") pod \"1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09\" (UID: \"1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09\") "
Sep 11 18:11:07.921174 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:07.921010    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09-metrics-certs\") pod \"1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09\" (UID: \"1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09\") "
Sep 11 18:11:07.921174 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:07.921033    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09-openshift-service-ca\") pod \"1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09\" (UID: \"1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09\") "
Sep 11 18:11:07.921542 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:07.921509    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09" (UID: "1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:11:07.923184 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:07.923150    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09" (UID: "1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:11:07.923302 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:07.923186    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09-kube-api-access-c5jdq" (OuterVolumeSpecName: "kube-api-access-c5jdq") pod "1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09" (UID: "1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09"). InnerVolumeSpecName "kube-api-access-c5jdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:11:07.923302 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:07.923193    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09" (UID: "1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:11:08.021945 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:08.021905    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09-metrics-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:11:08.021945 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:08.021940    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09-openshift-service-ca\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:11:08.021945 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:08.021949    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09-maas-api-tls\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:11:08.022180 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:08.021958    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5jdq\" (UniqueName: \"kubernetes.io/projected/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09-kube-api-access-c5jdq\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:11:08.464957 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:08.464923    2559 generic.go:356] "Generic (PLEG): container finished" podID="1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09" containerID="5f4d53ff4699c833ec6318ac3fcf68b4ae9afb3ac77203998ff28f2ab01b22c8" exitCode=0
Sep 11 18:11:08.465159 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:08.464975    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz"
Sep 11 18:11:08.465159 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:08.465039    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz" event={"ID":"1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09","Type":"ContainerDied","Data":"5f4d53ff4699c833ec6318ac3fcf68b4ae9afb3ac77203998ff28f2ab01b22c8"}
Sep 11 18:11:08.465159 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:08.465091    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz" event={"ID":"1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09","Type":"ContainerDied","Data":"f06faf337440564fd18cddc6b60d3abd6bc8057f247b876dee1ab5fa6d962c02"}
Sep 11 18:11:08.465159 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:08.465118    2559 scope.go:117] "RemoveContainer" containerID="5f4d53ff4699c833ec6318ac3fcf68b4ae9afb3ac77203998ff28f2ab01b22c8"
Sep 11 18:11:08.475871 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:08.475853    2559 scope.go:117] "RemoveContainer" containerID="5f4d53ff4699c833ec6318ac3fcf68b4ae9afb3ac77203998ff28f2ab01b22c8"
Sep 11 18:11:08.476187 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:11:08.476165    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f4d53ff4699c833ec6318ac3fcf68b4ae9afb3ac77203998ff28f2ab01b22c8\": container with ID starting with 5f4d53ff4699c833ec6318ac3fcf68b4ae9afb3ac77203998ff28f2ab01b22c8 not found: ID does not exist" containerID="5f4d53ff4699c833ec6318ac3fcf68b4ae9afb3ac77203998ff28f2ab01b22c8"
Sep 11 18:11:08.476263 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:08.476194    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f4d53ff4699c833ec6318ac3fcf68b4ae9afb3ac77203998ff28f2ab01b22c8"} err="failed to get container status \"5f4d53ff4699c833ec6318ac3fcf68b4ae9afb3ac77203998ff28f2ab01b22c8\": rpc error: code = NotFound desc = could not find container \"5f4d53ff4699c833ec6318ac3fcf68b4ae9afb3ac77203998ff28f2ab01b22c8\": container with ID starting with 5f4d53ff4699c833ec6318ac3fcf68b4ae9afb3ac77203998ff28f2ab01b22c8 not found: ID does not exist"
Sep 11 18:11:08.501050 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:08.501013    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz"]
Sep 11 18:11:08.506241 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:08.506214    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f95803-555fb5bd96-xs7lz"]
Sep 11 18:11:09.053390 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:09.053343    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09" path="/var/lib/kubelet/pods/1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09/volumes"
Sep 11 18:11:12.289176 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:12.289137    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"]
Sep 11 18:11:13.446884 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:13.446851    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"]
Sep 11 18:11:19.536337 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:19.536290    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:11:21.201947 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:21.201908    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq"]
Sep 11 18:11:21.202382 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:21.202284    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09" containerName="maas-api"
Sep 11 18:11:21.202382 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:21.202296    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09" containerName="maas-api"
Sep 11 18:11:21.202382 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:21.202307    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="472fa464-1370-4d6d-a851-dd1159a7708d" containerName="payload-processing"
Sep 11 18:11:21.202382 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:21.202313    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="472fa464-1370-4d6d-a851-dd1159a7708d" containerName="payload-processing"
Sep 11 18:11:21.202533 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:21.202391    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e8bd8df-3927-4ff7-ae3e-e1d9f8604e09" containerName="maas-api"
Sep 11 18:11:21.202533 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:21.202402    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="472fa464-1370-4d6d-a851-dd1159a7708d" containerName="payload-processing"
Sep 11 18:11:21.207013 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:21.206995    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq"
Sep 11 18:11:21.210174 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:21.210148    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6ad962-dockercfg-ssbzm\""
Sep 11 18:11:21.210174 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:21.210171    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6ad962\""
Sep 11 18:11:21.217062 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:21.217031    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq"]
Sep 11 18:11:21.358055 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:21.358013    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65t9s\" (UniqueName: \"kubernetes.io/projected/42e97169-c6ca-4575-8896-3f10f4f21836-kube-api-access-65t9s\") pod \"payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq\" (UID: \"42e97169-c6ca-4575-8896-3f10f4f21836\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq"
Sep 11 18:11:21.358258 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:21.358065    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e97169-c6ca-4575-8896-3f10f4f21836-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq\" (UID: \"42e97169-c6ca-4575-8896-3f10f4f21836\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq"
Sep 11 18:11:21.458940 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:21.458843    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65t9s\" (UniqueName: \"kubernetes.io/projected/42e97169-c6ca-4575-8896-3f10f4f21836-kube-api-access-65t9s\") pod \"payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq\" (UID: \"42e97169-c6ca-4575-8896-3f10f4f21836\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq"
Sep 11 18:11:21.458940 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:21.458892    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e97169-c6ca-4575-8896-3f10f4f21836-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq\" (UID: \"42e97169-c6ca-4575-8896-3f10f4f21836\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq"
Sep 11 18:11:21.459622 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:21.459602    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e97169-c6ca-4575-8896-3f10f4f21836-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq\" (UID: \"42e97169-c6ca-4575-8896-3f10f4f21836\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq"
Sep 11 18:11:21.471631 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:21.471606    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65t9s\" (UniqueName: \"kubernetes.io/projected/42e97169-c6ca-4575-8896-3f10f4f21836-kube-api-access-65t9s\") pod \"payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq\" (UID: \"42e97169-c6ca-4575-8896-3f10f4f21836\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq"
Sep 11 18:11:21.518437 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:21.518403    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq"
Sep 11 18:11:21.802972 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:21.802946    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq"]
Sep 11 18:11:21.804547 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:11:21.804519    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e97169_c6ca_4575_8896_3f10f4f21836.slice/crio-9aa0316d973031718369e6bbed908a0ae27f67bf66ae236a328e6618fbb6b20c WatchSource:0}: Error finding container 9aa0316d973031718369e6bbed908a0ae27f67bf66ae236a328e6618fbb6b20c: Status 404 returned error can't find the container with id 9aa0316d973031718369e6bbed908a0ae27f67bf66ae236a328e6618fbb6b20c
Sep 11 18:11:22.542599 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:22.542560    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq" event={"ID":"42e97169-c6ca-4575-8896-3f10f4f21836","Type":"ContainerStarted","Data":"9e630cbceebfd0d50e9b727b3f6f561bc492028c746b2901e2377fafb40fd4fa"}
Sep 11 18:11:22.543025 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:22.542617    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq" event={"ID":"42e97169-c6ca-4575-8896-3f10f4f21836","Type":"ContainerStarted","Data":"9aa0316d973031718369e6bbed908a0ae27f67bf66ae236a328e6618fbb6b20c"}
Sep 11 18:11:22.543025 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:22.542662    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq"
Sep 11 18:11:22.576283 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:22.576226    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq" podStartSLOduration=1.5762082099999999 podStartE2EDuration="1.57620821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:11:22.575203518 +0000 UTC m=+1588.020304113" watchObservedRunningTime="2026-09-11 18:11:22.57620821 +0000 UTC m=+1588.021308780"
Sep 11 18:11:27.308196 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.308132    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m" podUID="27f7a2e9-07ef-4da5-835d-0ede48937e49" containerName="main" containerID="cri-o://d24cdcfbb8f39b6f0a6a1d9af947e95defcc02ce5cb6a85a1661bf4cd27af1bc" gracePeriod=45
Sep 11 18:11:27.548983 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.548957    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:11:27.574804 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.574707    2559 generic.go:356] "Generic (PLEG): container finished" podID="27f7a2e9-07ef-4da5-835d-0ede48937e49" containerID="d24cdcfbb8f39b6f0a6a1d9af947e95defcc02ce5cb6a85a1661bf4cd27af1bc" exitCode=0
Sep 11 18:11:27.574804 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.574752    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"
Sep 11 18:11:27.574804 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.574755    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m" event={"ID":"27f7a2e9-07ef-4da5-835d-0ede48937e49","Type":"ContainerDied","Data":"d24cdcfbb8f39b6f0a6a1d9af947e95defcc02ce5cb6a85a1661bf4cd27af1bc"}
Sep 11 18:11:27.574804 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.574794    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m" event={"ID":"27f7a2e9-07ef-4da5-835d-0ede48937e49","Type":"ContainerDied","Data":"a810124cce75c660c1fefa41ff4ef9ee2ae82765385c85f3b91e0d9cfbe32a00"}
Sep 11 18:11:27.575151 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.574817    2559 scope.go:117] "RemoveContainer" containerID="d24cdcfbb8f39b6f0a6a1d9af947e95defcc02ce5cb6a85a1661bf4cd27af1bc"
Sep 11 18:11:27.584775 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.584749    2559 scope.go:117] "RemoveContainer" containerID="d24cdcfbb8f39b6f0a6a1d9af947e95defcc02ce5cb6a85a1661bf4cd27af1bc"
Sep 11 18:11:27.585061 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:11:27.585036    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d24cdcfbb8f39b6f0a6a1d9af947e95defcc02ce5cb6a85a1661bf4cd27af1bc\": container with ID starting with d24cdcfbb8f39b6f0a6a1d9af947e95defcc02ce5cb6a85a1661bf4cd27af1bc not found: ID does not exist" containerID="d24cdcfbb8f39b6f0a6a1d9af947e95defcc02ce5cb6a85a1661bf4cd27af1bc"
Sep 11 18:11:27.585186 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.585069    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d24cdcfbb8f39b6f0a6a1d9af947e95defcc02ce5cb6a85a1661bf4cd27af1bc"} err="failed to get container status \"d24cdcfbb8f39b6f0a6a1d9af947e95defcc02ce5cb6a85a1661bf4cd27af1bc\": rpc error: code = NotFound desc = could not find container \"d24cdcfbb8f39b6f0a6a1d9af947e95defcc02ce5cb6a85a1661bf4cd27af1bc\": container with ID starting with d24cdcfbb8f39b6f0a6a1d9af947e95defcc02ce5cb6a85a1661bf4cd27af1bc not found: ID does not exist"
Sep 11 18:11:27.721289 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.721255    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f7a2e9-07ef-4da5-835d-0ede48937e49-model-cache\") pod \"27f7a2e9-07ef-4da5-835d-0ede48937e49\" (UID: \"27f7a2e9-07ef-4da5-835d-0ede48937e49\") "
Sep 11 18:11:27.721466 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.721310    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f7a2e9-07ef-4da5-835d-0ede48937e49-tls-certs\") pod \"27f7a2e9-07ef-4da5-835d-0ede48937e49\" (UID: \"27f7a2e9-07ef-4da5-835d-0ede48937e49\") "
Sep 11 18:11:27.721466 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.721391    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f7a2e9-07ef-4da5-835d-0ede48937e49-home\") pod \"27f7a2e9-07ef-4da5-835d-0ede48937e49\" (UID: \"27f7a2e9-07ef-4da5-835d-0ede48937e49\") "
Sep 11 18:11:27.721466 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.721433    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f7a2e9-07ef-4da5-835d-0ede48937e49-tmp-dir\") pod \"27f7a2e9-07ef-4da5-835d-0ede48937e49\" (UID: \"27f7a2e9-07ef-4da5-835d-0ede48937e49\") "
Sep 11 18:11:27.721614 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.721481    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f7a2e9-07ef-4da5-835d-0ede48937e49-dshm\") pod \"27f7a2e9-07ef-4da5-835d-0ede48937e49\" (UID: \"27f7a2e9-07ef-4da5-835d-0ede48937e49\") "
Sep 11 18:11:27.721614 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.721524    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmvtv\" (UniqueName: \"kubernetes.io/projected/27f7a2e9-07ef-4da5-835d-0ede48937e49-kube-api-access-jmvtv\") pod \"27f7a2e9-07ef-4da5-835d-0ede48937e49\" (UID: \"27f7a2e9-07ef-4da5-835d-0ede48937e49\") "
Sep 11 18:11:27.721716 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.721631    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f7a2e9-07ef-4da5-835d-0ede48937e49-model-cache" (OuterVolumeSpecName: "model-cache") pod "27f7a2e9-07ef-4da5-835d-0ede48937e49" (UID: "27f7a2e9-07ef-4da5-835d-0ede48937e49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:11:27.721774 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.721719    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f7a2e9-07ef-4da5-835d-0ede48937e49-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "27f7a2e9-07ef-4da5-835d-0ede48937e49" (UID: "27f7a2e9-07ef-4da5-835d-0ede48937e49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:11:27.721774 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.721730    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f7a2e9-07ef-4da5-835d-0ede48937e49-home" (OuterVolumeSpecName: "home") pod "27f7a2e9-07ef-4da5-835d-0ede48937e49" (UID: "27f7a2e9-07ef-4da5-835d-0ede48937e49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:11:27.721916 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.721893    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f7a2e9-07ef-4da5-835d-0ede48937e49-model-cache\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:11:27.721916 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.721916    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f7a2e9-07ef-4da5-835d-0ede48937e49-home\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:11:27.722105 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.721943    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f7a2e9-07ef-4da5-835d-0ede48937e49-tmp-dir\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:11:27.723639 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.723609    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f7a2e9-07ef-4da5-835d-0ede48937e49-dshm" (OuterVolumeSpecName: "dshm") pod "27f7a2e9-07ef-4da5-835d-0ede48937e49" (UID: "27f7a2e9-07ef-4da5-835d-0ede48937e49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:11:27.723639 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.723626    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f7a2e9-07ef-4da5-835d-0ede48937e49-kube-api-access-jmvtv" (OuterVolumeSpecName: "kube-api-access-jmvtv") pod "27f7a2e9-07ef-4da5-835d-0ede48937e49" (UID: "27f7a2e9-07ef-4da5-835d-0ede48937e49"). InnerVolumeSpecName "kube-api-access-jmvtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:11:27.723639 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.723620    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f7a2e9-07ef-4da5-835d-0ede48937e49-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27f7a2e9-07ef-4da5-835d-0ede48937e49" (UID: "27f7a2e9-07ef-4da5-835d-0ede48937e49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:11:27.822955 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.822911    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f7a2e9-07ef-4da5-835d-0ede48937e49-dshm\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:11:27.822955 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.822946    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmvtv\" (UniqueName: \"kubernetes.io/projected/27f7a2e9-07ef-4da5-835d-0ede48937e49-kube-api-access-jmvtv\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:11:27.822955 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.822960    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f7a2e9-07ef-4da5-835d-0ede48937e49-tls-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:11:27.897606 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.897570    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"]
Sep 11 18:11:27.902133 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:27.902104    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-386935/rate-test-model-386935-kserve-6fdccf99c9-f765m"]
Sep 11 18:11:28.465991 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:28.465923    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82" podUID="d4bbb691-9523-4c4b-810b-c9dec3424c07" containerName="main" containerID="cri-o://b0a83488df15ae541bf350a037fefa3bb317b413d11d4d90f0d868be06964e72" gracePeriod=45
Sep 11 18:11:28.726830 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:28.726759    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:11:28.832048 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:28.832015    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4bbb691-9523-4c4b-810b-c9dec3424c07-tmp-dir\") pod \"d4bbb691-9523-4c4b-810b-c9dec3424c07\" (UID: \"d4bbb691-9523-4c4b-810b-c9dec3424c07\") "
Sep 11 18:11:28.832208 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:28.832075    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4bbb691-9523-4c4b-810b-c9dec3424c07-model-cache\") pod \"d4bbb691-9523-4c4b-810b-c9dec3424c07\" (UID: \"d4bbb691-9523-4c4b-810b-c9dec3424c07\") "
Sep 11 18:11:28.832208 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:28.832128    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6q6d\" (UniqueName: \"kubernetes.io/projected/d4bbb691-9523-4c4b-810b-c9dec3424c07-kube-api-access-d6q6d\") pod \"d4bbb691-9523-4c4b-810b-c9dec3424c07\" (UID: \"d4bbb691-9523-4c4b-810b-c9dec3424c07\") "
Sep 11 18:11:28.832208 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:28.832186    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4bbb691-9523-4c4b-810b-c9dec3424c07-dshm\") pod \"d4bbb691-9523-4c4b-810b-c9dec3424c07\" (UID: \"d4bbb691-9523-4c4b-810b-c9dec3424c07\") "
Sep 11 18:11:28.832409 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:28.832234    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4bbb691-9523-4c4b-810b-c9dec3424c07-home\") pod \"d4bbb691-9523-4c4b-810b-c9dec3424c07\" (UID: \"d4bbb691-9523-4c4b-810b-c9dec3424c07\") "
Sep 11 18:11:28.832409 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:28.832277    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4bbb691-9523-4c4b-810b-c9dec3424c07-tls-certs\") pod \"d4bbb691-9523-4c4b-810b-c9dec3424c07\" (UID: \"d4bbb691-9523-4c4b-810b-c9dec3424c07\") "
Sep 11 18:11:28.832409 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:28.832309    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4bbb691-9523-4c4b-810b-c9dec3424c07-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4bbb691-9523-4c4b-810b-c9dec3424c07" (UID: "d4bbb691-9523-4c4b-810b-c9dec3424c07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:11:28.832586 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:28.832424    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4bbb691-9523-4c4b-810b-c9dec3424c07-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4bbb691-9523-4c4b-810b-c9dec3424c07" (UID: "d4bbb691-9523-4c4b-810b-c9dec3424c07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:11:28.832586 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:28.832507    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4bbb691-9523-4c4b-810b-c9dec3424c07-home" (OuterVolumeSpecName: "home") pod "d4bbb691-9523-4c4b-810b-c9dec3424c07" (UID: "d4bbb691-9523-4c4b-810b-c9dec3424c07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:11:28.832690 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:28.832632    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4bbb691-9523-4c4b-810b-c9dec3424c07-model-cache\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:11:28.832690 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:28.832647    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4bbb691-9523-4c4b-810b-c9dec3424c07-home\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:11:28.832690 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:28.832661    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4bbb691-9523-4c4b-810b-c9dec3424c07-tmp-dir\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:11:28.834511 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:28.834476    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4bbb691-9523-4c4b-810b-c9dec3424c07-dshm" (OuterVolumeSpecName: "dshm") pod "d4bbb691-9523-4c4b-810b-c9dec3424c07" (UID: "d4bbb691-9523-4c4b-810b-c9dec3424c07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:11:28.834640 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:28.834524    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4bbb691-9523-4c4b-810b-c9dec3424c07-kube-api-access-d6q6d" (OuterVolumeSpecName: "kube-api-access-d6q6d") pod "d4bbb691-9523-4c4b-810b-c9dec3424c07" (UID: "d4bbb691-9523-4c4b-810b-c9dec3424c07"). InnerVolumeSpecName "kube-api-access-d6q6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:11:28.834640 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:28.834557    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4bbb691-9523-4c4b-810b-c9dec3424c07-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4bbb691-9523-4c4b-810b-c9dec3424c07" (UID: "d4bbb691-9523-4c4b-810b-c9dec3424c07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:11:28.933999 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:28.933961    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6q6d\" (UniqueName: \"kubernetes.io/projected/d4bbb691-9523-4c4b-810b-c9dec3424c07-kube-api-access-d6q6d\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:11:28.933999 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:28.933994    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4bbb691-9523-4c4b-810b-c9dec3424c07-dshm\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:11:28.933999 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:28.934005    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4bbb691-9523-4c4b-810b-c9dec3424c07-tls-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:11:29.053591 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:29.053509    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27f7a2e9-07ef-4da5-835d-0ede48937e49" path="/var/lib/kubelet/pods/27f7a2e9-07ef-4da5-835d-0ede48937e49/volumes"
Sep 11 18:11:29.588225 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:29.588189    2559 generic.go:356] "Generic (PLEG): container finished" podID="d4bbb691-9523-4c4b-810b-c9dec3424c07" containerID="b0a83488df15ae541bf350a037fefa3bb317b413d11d4d90f0d868be06964e72" exitCode=0
Sep 11 18:11:29.588727 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:29.588233    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"
Sep 11 18:11:29.588727 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:29.588277    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82" event={"ID":"d4bbb691-9523-4c4b-810b-c9dec3424c07","Type":"ContainerDied","Data":"b0a83488df15ae541bf350a037fefa3bb317b413d11d4d90f0d868be06964e72"}
Sep 11 18:11:29.588727 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:29.588326    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82" event={"ID":"d4bbb691-9523-4c4b-810b-c9dec3424c07","Type":"ContainerDied","Data":"c2a29378ace99331bace805f3ea8437069f047add6dc7ed6ce1160382ec0337f"}
Sep 11 18:11:29.588727 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:29.588345    2559 scope.go:117] "RemoveContainer" containerID="b0a83488df15ae541bf350a037fefa3bb317b413d11d4d90f0d868be06964e72"
Sep 11 18:11:29.597109 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:29.597089    2559 scope.go:117] "RemoveContainer" containerID="b0a83488df15ae541bf350a037fefa3bb317b413d11d4d90f0d868be06964e72"
Sep 11 18:11:29.597396 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:11:29.597377    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0a83488df15ae541bf350a037fefa3bb317b413d11d4d90f0d868be06964e72\": container with ID starting with b0a83488df15ae541bf350a037fefa3bb317b413d11d4d90f0d868be06964e72 not found: ID does not exist" containerID="b0a83488df15ae541bf350a037fefa3bb317b413d11d4d90f0d868be06964e72"
Sep 11 18:11:29.597454 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:29.597404    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a83488df15ae541bf350a037fefa3bb317b413d11d4d90f0d868be06964e72"} err="failed to get container status \"b0a83488df15ae541bf350a037fefa3bb317b413d11d4d90f0d868be06964e72\": rpc error: code = NotFound desc = could not find container \"b0a83488df15ae541bf350a037fefa3bb317b413d11d4d90f0d868be06964e72\": container with ID starting with b0a83488df15ae541bf350a037fefa3bb317b413d11d4d90f0d868be06964e72 not found: ID does not exist"
Sep 11 18:11:29.609713 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:29.609689    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"]
Sep 11 18:11:29.611917 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:29.611894    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3ebea5/rate-test-model-3ebea5-kserve-9d4bc-wlp82"]
Sep 11 18:11:30.135711 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:30.135671    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:11:31.053345 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:31.053306    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4bbb691-9523-4c4b-810b-c9dec3424c07" path="/var/lib/kubelet/pods/d4bbb691-9523-4c4b-810b-c9dec3424c07/volumes"
Sep 11 18:11:33.550805 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:33.550770    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq"
Sep 11 18:11:55.331498 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.331472    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn"]
Sep 11 18:11:55.331838 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.331828    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4bbb691-9523-4c4b-810b-c9dec3424c07" containerName="main"
Sep 11 18:11:55.331882 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.331839    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4bbb691-9523-4c4b-810b-c9dec3424c07" containerName="main"
Sep 11 18:11:55.331882 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.331863    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27f7a2e9-07ef-4da5-835d-0ede48937e49" containerName="main"
Sep 11 18:11:55.331882 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.331871    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="27f7a2e9-07ef-4da5-835d-0ede48937e49" containerName="main"
Sep 11 18:11:55.331982 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.331929    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="27f7a2e9-07ef-4da5-835d-0ede48937e49" containerName="main"
Sep 11 18:11:55.331982 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.331937    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4bbb691-9523-4c4b-810b-c9dec3424c07" containerName="main"
Sep 11 18:11:55.336520 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.336501    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn"
Sep 11 18:11:55.341250 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.341226    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b54eb6-serving-cert\""
Sep 11 18:11:55.447798 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.447765    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn"]
Sep 11 18:11:55.489573 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.489544    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c628c036-8daf-4e06-ac35-e6506a639f33-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn\" (UID: \"c628c036-8daf-4e06-ac35-e6506a639f33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn"
Sep 11 18:11:55.489791 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.489607    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c628c036-8daf-4e06-ac35-e6506a639f33-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn\" (UID: \"c628c036-8daf-4e06-ac35-e6506a639f33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn"
Sep 11 18:11:55.489791 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.489634    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrps8\" (UniqueName: \"kubernetes.io/projected/c628c036-8daf-4e06-ac35-e6506a639f33-kube-api-access-vrps8\") pod \"maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn\" (UID: \"c628c036-8daf-4e06-ac35-e6506a639f33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn"
Sep 11 18:11:55.489791 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.489735    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c628c036-8daf-4e06-ac35-e6506a639f33-metrics-certs\") pod \"maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn\" (UID: \"c628c036-8daf-4e06-ac35-e6506a639f33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn"
Sep 11 18:11:55.590530 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.590438    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c628c036-8daf-4e06-ac35-e6506a639f33-metrics-certs\") pod \"maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn\" (UID: \"c628c036-8daf-4e06-ac35-e6506a639f33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn"
Sep 11 18:11:55.590530 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.590500    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c628c036-8daf-4e06-ac35-e6506a639f33-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn\" (UID: \"c628c036-8daf-4e06-ac35-e6506a639f33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn"
Sep 11 18:11:55.590735 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.590535    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c628c036-8daf-4e06-ac35-e6506a639f33-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn\" (UID: \"c628c036-8daf-4e06-ac35-e6506a639f33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn"
Sep 11 18:11:55.590735 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.590556    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrps8\" (UniqueName: \"kubernetes.io/projected/c628c036-8daf-4e06-ac35-e6506a639f33-kube-api-access-vrps8\") pod \"maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn\" (UID: \"c628c036-8daf-4e06-ac35-e6506a639f33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn"
Sep 11 18:11:55.591275 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.591243    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c628c036-8daf-4e06-ac35-e6506a639f33-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn\" (UID: \"c628c036-8daf-4e06-ac35-e6506a639f33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn"
Sep 11 18:11:55.593120 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.593092    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c628c036-8daf-4e06-ac35-e6506a639f33-metrics-certs\") pod \"maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn\" (UID: \"c628c036-8daf-4e06-ac35-e6506a639f33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn"
Sep 11 18:11:55.593237 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.593126    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c628c036-8daf-4e06-ac35-e6506a639f33-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn\" (UID: \"c628c036-8daf-4e06-ac35-e6506a639f33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn"
Sep 11 18:11:55.616875 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.616846    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrps8\" (UniqueName: \"kubernetes.io/projected/c628c036-8daf-4e06-ac35-e6506a639f33-kube-api-access-vrps8\") pod \"maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn\" (UID: \"c628c036-8daf-4e06-ac35-e6506a639f33\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn"
Sep 11 18:11:55.646819 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.646776    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn"
Sep 11 18:11:55.936878 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.936849    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn"]
Sep 11 18:11:55.938835 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:11:55.938808    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc628c036_8daf_4e06_ac35_e6506a639f33.slice/crio-eef04d3002161a55798b8dfe94037e85c4aea1fe3b900eb39b9564ddfa05228f WatchSource:0}: Error finding container eef04d3002161a55798b8dfe94037e85c4aea1fe3b900eb39b9564ddfa05228f: Status 404 returned error can't find the container with id eef04d3002161a55798b8dfe94037e85c4aea1fe3b900eb39b9564ddfa05228f
Sep 11 18:11:55.940262 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:55.940246    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:11:56.652974 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:56.652940    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95"]
Sep 11 18:11:56.656280 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:56.656261    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95"
Sep 11 18:11:56.659985 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:56.659964    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b54eb6-dockercfg-5vvww\""
Sep 11 18:11:56.661081 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:56.661062    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b54eb6\""
Sep 11 18:11:56.706200 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:56.706164    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95"]
Sep 11 18:11:56.746528 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:56.746486    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn" event={"ID":"c628c036-8daf-4e06-ac35-e6506a639f33","Type":"ContainerStarted","Data":"fd1b8fcf842ec46d7c8905e5a55795b27dc0e58c5655f2ace56cd43f94e425ec"}
Sep 11 18:11:56.746708 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:56.746542    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn" event={"ID":"c628c036-8daf-4e06-ac35-e6506a639f33","Type":"ContainerStarted","Data":"eef04d3002161a55798b8dfe94037e85c4aea1fe3b900eb39b9564ddfa05228f"}
Sep 11 18:11:56.746708 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:56.746564    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn"
Sep 11 18:11:56.803226 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:56.803184    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn8xb\" (UniqueName: \"kubernetes.io/projected/4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb-kube-api-access-gn8xb\") pod \"payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95\" (UID: \"4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95"
Sep 11 18:11:56.803432 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:56.803321    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95\" (UID: \"4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95"
Sep 11 18:11:56.827901 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:56.827845    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn" podStartSLOduration=1.174928061 podStartE2EDuration="1.827822104s" totalImagesPullingTime="652.894043ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:11:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:11:56.825319883 +0000 UTC m=+1622.270420479" watchObservedRunningTime="2026-09-11 18:11:56.827822104 +0000 UTC m=+1622.272922681"
Sep 11 18:11:56.903922 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:56.903881    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95\" (UID: \"4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95"
Sep 11 18:11:56.904130 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:56.903986    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn8xb\" (UniqueName: \"kubernetes.io/projected/4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb-kube-api-access-gn8xb\") pod \"payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95\" (UID: \"4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95"
Sep 11 18:11:56.904733 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:56.904710    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95\" (UID: \"4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95"
Sep 11 18:11:56.925991 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:56.925958    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn8xb\" (UniqueName: \"kubernetes.io/projected/4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb-kube-api-access-gn8xb\") pod \"payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95\" (UID: \"4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95"
Sep 11 18:11:56.970055 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:56.970013    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95"
Sep 11 18:11:57.280161 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:57.280066    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95"]
Sep 11 18:11:57.287057 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:11:57.287029    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd64dd4_583b_4f3a_8bfa_3c7f94e777eb.slice/crio-2ec1739f785311bf3282ceb1fd037ff50bf7eff982a13391eb6c75091e67c425 WatchSource:0}: Error finding container 2ec1739f785311bf3282ceb1fd037ff50bf7eff982a13391eb6c75091e67c425: Status 404 returned error can't find the container with id 2ec1739f785311bf3282ceb1fd037ff50bf7eff982a13391eb6c75091e67c425
Sep 11 18:11:57.753099 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:57.753050    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95" event={"ID":"4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb","Type":"ContainerStarted","Data":"fff8ccbb1017a38db8a1b90c129ac745a1dc000065d65f53279780e1dc00d437"}
Sep 11 18:11:57.753611 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:57.753123    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95" event={"ID":"4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb","Type":"ContainerStarted","Data":"2ec1739f785311bf3282ceb1fd037ff50bf7eff982a13391eb6c75091e67c425"}
Sep 11 18:11:57.753611 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:57.753303    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95"
Sep 11 18:11:57.794682 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:11:57.794625    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95" podStartSLOduration=1.794607737 podStartE2EDuration="1.794607737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:11:57.794246422 +0000 UTC m=+1623.239346995" watchObservedRunningTime="2026-09-11 18:11:57.794607737 +0000 UTC m=+1623.239708307"
Sep 11 18:12:00.408910 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:00.408873    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:12:02.758777 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:02.758747    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn"
Sep 11 18:12:08.760360 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:08.760326    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95"
Sep 11 18:12:27.022671 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:27.022639    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95_4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb/payload-processing/0.log"
Sep 11 18:12:55.359908 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.359875    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"]
Sep 11 18:12:55.364553 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.364533    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:12:55.368133 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.368106    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-6ad962\"/\"ipp-route-6ad962-kserve-self-signed-certs\""
Sep 11 18:12:55.368277 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.368145    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-6ad962\"/\"default-dockercfg-plc6c\""
Sep 11 18:12:55.368277 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.368188    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-6ad962\"/\"kube-root-ca.crt\""
Sep 11 18:12:55.368466 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.368287    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-6ad962\"/\"openshift-service-ca.crt\""
Sep 11 18:12:55.374317 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.374292    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"]
Sep 11 18:12:55.472518 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.472470    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/998837d8-d744-40bf-98f6-eed6a4ccec08-home\") pod \"ipp-route-6ad962-kserve-679554c455-s95pl\" (UID: \"998837d8-d744-40bf-98f6-eed6a4ccec08\") " pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:12:55.472701 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.472527    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22d8j\" (UniqueName: \"kubernetes.io/projected/998837d8-d744-40bf-98f6-eed6a4ccec08-kube-api-access-22d8j\") pod \"ipp-route-6ad962-kserve-679554c455-s95pl\" (UID: \"998837d8-d744-40bf-98f6-eed6a4ccec08\") " pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:12:55.472701 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.472587    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/998837d8-d744-40bf-98f6-eed6a4ccec08-model-cache\") pod \"ipp-route-6ad962-kserve-679554c455-s95pl\" (UID: \"998837d8-d744-40bf-98f6-eed6a4ccec08\") " pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:12:55.472701 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.472609    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/998837d8-d744-40bf-98f6-eed6a4ccec08-tmp-dir\") pod \"ipp-route-6ad962-kserve-679554c455-s95pl\" (UID: \"998837d8-d744-40bf-98f6-eed6a4ccec08\") " pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:12:55.472701 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.472665    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/998837d8-d744-40bf-98f6-eed6a4ccec08-tls-certs\") pod \"ipp-route-6ad962-kserve-679554c455-s95pl\" (UID: \"998837d8-d744-40bf-98f6-eed6a4ccec08\") " pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:12:55.472848 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.472731    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/998837d8-d744-40bf-98f6-eed6a4ccec08-dshm\") pod \"ipp-route-6ad962-kserve-679554c455-s95pl\" (UID: \"998837d8-d744-40bf-98f6-eed6a4ccec08\") " pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:12:55.573949 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.573905    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/998837d8-d744-40bf-98f6-eed6a4ccec08-home\") pod \"ipp-route-6ad962-kserve-679554c455-s95pl\" (UID: \"998837d8-d744-40bf-98f6-eed6a4ccec08\") " pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:12:55.574140 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.573957    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22d8j\" (UniqueName: \"kubernetes.io/projected/998837d8-d744-40bf-98f6-eed6a4ccec08-kube-api-access-22d8j\") pod \"ipp-route-6ad962-kserve-679554c455-s95pl\" (UID: \"998837d8-d744-40bf-98f6-eed6a4ccec08\") " pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:12:55.574140 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.574004    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/998837d8-d744-40bf-98f6-eed6a4ccec08-model-cache\") pod \"ipp-route-6ad962-kserve-679554c455-s95pl\" (UID: \"998837d8-d744-40bf-98f6-eed6a4ccec08\") " pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:12:55.574140 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.574033    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/998837d8-d744-40bf-98f6-eed6a4ccec08-tmp-dir\") pod \"ipp-route-6ad962-kserve-679554c455-s95pl\" (UID: \"998837d8-d744-40bf-98f6-eed6a4ccec08\") " pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:12:55.574140 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.574072    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/998837d8-d744-40bf-98f6-eed6a4ccec08-tls-certs\") pod \"ipp-route-6ad962-kserve-679554c455-s95pl\" (UID: \"998837d8-d744-40bf-98f6-eed6a4ccec08\") " pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:12:55.574140 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.574122    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/998837d8-d744-40bf-98f6-eed6a4ccec08-dshm\") pod \"ipp-route-6ad962-kserve-679554c455-s95pl\" (UID: \"998837d8-d744-40bf-98f6-eed6a4ccec08\") " pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:12:55.574446 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.574415    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/998837d8-d744-40bf-98f6-eed6a4ccec08-home\") pod \"ipp-route-6ad962-kserve-679554c455-s95pl\" (UID: \"998837d8-d744-40bf-98f6-eed6a4ccec08\") " pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:12:55.574508 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.574451    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/998837d8-d744-40bf-98f6-eed6a4ccec08-tmp-dir\") pod \"ipp-route-6ad962-kserve-679554c455-s95pl\" (UID: \"998837d8-d744-40bf-98f6-eed6a4ccec08\") " pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:12:55.574508 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.574482    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/998837d8-d744-40bf-98f6-eed6a4ccec08-model-cache\") pod \"ipp-route-6ad962-kserve-679554c455-s95pl\" (UID: \"998837d8-d744-40bf-98f6-eed6a4ccec08\") " pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:12:55.576475 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.576452    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/998837d8-d744-40bf-98f6-eed6a4ccec08-dshm\") pod \"ipp-route-6ad962-kserve-679554c455-s95pl\" (UID: \"998837d8-d744-40bf-98f6-eed6a4ccec08\") " pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:12:55.576782 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.576762    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/998837d8-d744-40bf-98f6-eed6a4ccec08-tls-certs\") pod \"ipp-route-6ad962-kserve-679554c455-s95pl\" (UID: \"998837d8-d744-40bf-98f6-eed6a4ccec08\") " pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:12:55.583708 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.583685    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22d8j\" (UniqueName: \"kubernetes.io/projected/998837d8-d744-40bf-98f6-eed6a4ccec08-kube-api-access-22d8j\") pod \"ipp-route-6ad962-kserve-679554c455-s95pl\" (UID: \"998837d8-d744-40bf-98f6-eed6a4ccec08\") " pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:12:55.674965 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.674921    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:12:55.948598 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:55.948565    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"]
Sep 11 18:12:55.949975 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:12:55.949938    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod998837d8_d744_40bf_98f6_eed6a4ccec08.slice/crio-6e7a3894e76ecc918726a6ea1815a3864637438822aaae88d4718ac352c58469 WatchSource:0}: Error finding container 6e7a3894e76ecc918726a6ea1815a3864637438822aaae88d4718ac352c58469: Status 404 returned error can't find the container with id 6e7a3894e76ecc918726a6ea1815a3864637438822aaae88d4718ac352c58469
Sep 11 18:12:56.071323 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:56.071285    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl" event={"ID":"998837d8-d744-40bf-98f6-eed6a4ccec08","Type":"ContainerStarted","Data":"01d2c91c74fe50b52969a33cc43badd6b0ed7823bc091a8dd7ce17b651a769f8"}
Sep 11 18:12:56.071517 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:56.071334    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl" event={"ID":"998837d8-d744-40bf-98f6-eed6a4ccec08","Type":"ContainerStarted","Data":"6e7a3894e76ecc918726a6ea1815a3864637438822aaae88d4718ac352c58469"}
Sep 11 18:12:56.091823 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:56.091770    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl" podStartSLOduration=1.091753654 podStartE2EDuration="1.091753654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:12:56.089423289 +0000 UTC m=+1681.534523859" watchObservedRunningTime="2026-09-11 18:12:56.091753654 +0000 UTC m=+1681.536854225"
Sep 11 18:12:59.238005 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:12:59.237960    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:13:05.675451 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:05.675411    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:13:05.675983 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:05.675476    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:13:05.688594 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:05.688564    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:13:06.134822 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:06.134788    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:13:16.937980 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:16.937939    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:13:23.532010 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:23.531977    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95_4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb/payload-processing/0.log"
Sep 11 18:13:31.334387 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:31.334330    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx"]
Sep 11 18:13:31.337862 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:31.337843    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx"
Sep 11 18:13:31.340929 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:31.340906    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-4f951a\""
Sep 11 18:13:31.350465 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:31.350435    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx"]
Sep 11 18:13:31.354255 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:31.354218    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g"]
Sep 11 18:13:31.357841 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:31.357825    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g"
Sep 11 18:13:31.360621 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:31.360596    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-4f951a-dockercfg-h48qv\""
Sep 11 18:13:31.370610 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:31.370586    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g"]
Sep 11 18:13:31.405861 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:31.405825    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b41d347-0b15-498c-b44b-6771e8ced079-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx\" (UID: \"0b41d347-0b15-498c-b44b-6771e8ced079\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx"
Sep 11 18:13:31.406036 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:31.405918    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ztdp\" (UniqueName: \"kubernetes.io/projected/4d67ff0b-8104-45d1-8eee-49b41fb9edf9-kube-api-access-7ztdp\") pod \"payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g\" (UID: \"4d67ff0b-8104-45d1-8eee-49b41fb9edf9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g"
Sep 11 18:13:31.406036 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:31.405941    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d67ff0b-8104-45d1-8eee-49b41fb9edf9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g\" (UID: \"4d67ff0b-8104-45d1-8eee-49b41fb9edf9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g"
Sep 11 18:13:31.406036 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:31.405960    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d99xp\" (UniqueName: \"kubernetes.io/projected/0b41d347-0b15-498c-b44b-6771e8ced079-kube-api-access-d99xp\") pod \"payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx\" (UID: \"0b41d347-0b15-498c-b44b-6771e8ced079\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx"
Sep 11 18:13:31.507277 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:31.507235    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ztdp\" (UniqueName: \"kubernetes.io/projected/4d67ff0b-8104-45d1-8eee-49b41fb9edf9-kube-api-access-7ztdp\") pod \"payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g\" (UID: \"4d67ff0b-8104-45d1-8eee-49b41fb9edf9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g"
Sep 11 18:13:31.507277 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:31.507274    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d67ff0b-8104-45d1-8eee-49b41fb9edf9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g\" (UID: \"4d67ff0b-8104-45d1-8eee-49b41fb9edf9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g"
Sep 11 18:13:31.507560 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:31.507296    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d99xp\" (UniqueName: \"kubernetes.io/projected/0b41d347-0b15-498c-b44b-6771e8ced079-kube-api-access-d99xp\") pod \"payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx\" (UID: \"0b41d347-0b15-498c-b44b-6771e8ced079\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx"
Sep 11 18:13:31.507560 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:31.507399    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b41d347-0b15-498c-b44b-6771e8ced079-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx\" (UID: \"0b41d347-0b15-498c-b44b-6771e8ced079\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx"
Sep 11 18:13:31.508062 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:31.508037    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d67ff0b-8104-45d1-8eee-49b41fb9edf9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g\" (UID: \"4d67ff0b-8104-45d1-8eee-49b41fb9edf9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g"
Sep 11 18:13:31.508183 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:31.508156    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b41d347-0b15-498c-b44b-6771e8ced079-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx\" (UID: \"0b41d347-0b15-498c-b44b-6771e8ced079\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx"
Sep 11 18:13:31.516256 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:31.516226    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d99xp\" (UniqueName: \"kubernetes.io/projected/0b41d347-0b15-498c-b44b-6771e8ced079-kube-api-access-d99xp\") pod \"payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx\" (UID: \"0b41d347-0b15-498c-b44b-6771e8ced079\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx"
Sep 11 18:13:31.516407 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:31.516275    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ztdp\" (UniqueName: \"kubernetes.io/projected/4d67ff0b-8104-45d1-8eee-49b41fb9edf9-kube-api-access-7ztdp\") pod \"payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g\" (UID: \"4d67ff0b-8104-45d1-8eee-49b41fb9edf9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g"
Sep 11 18:13:31.648343 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:31.648303    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx"
Sep 11 18:13:31.668578 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:31.668546    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g"
Sep 11 18:13:31.923643 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:31.923615    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx"]
Sep 11 18:13:31.925316 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:13:31.925293    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b41d347_0b15_498c_b44b_6771e8ced079.slice/crio-e179583e9297a8fb8c4c5e4da5ae9f223c91aa05e4641be47989c1e7ff742029 WatchSource:0}: Error finding container e179583e9297a8fb8c4c5e4da5ae9f223c91aa05e4641be47989c1e7ff742029: Status 404 returned error can't find the container with id e179583e9297a8fb8c4c5e4da5ae9f223c91aa05e4641be47989c1e7ff742029
Sep 11 18:13:31.956359 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:31.956311    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g"]
Sep 11 18:13:31.961585 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:13:31.961555    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d67ff0b_8104_45d1_8eee_49b41fb9edf9.slice/crio-a7a02dd26df05257be1fd321da6f24901a6d12ba77dbfa220b64102866a4874d WatchSource:0}: Error finding container a7a02dd26df05257be1fd321da6f24901a6d12ba77dbfa220b64102866a4874d: Status 404 returned error can't find the container with id a7a02dd26df05257be1fd321da6f24901a6d12ba77dbfa220b64102866a4874d
Sep 11 18:13:32.269292 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:32.269185    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx" event={"ID":"0b41d347-0b15-498c-b44b-6771e8ced079","Type":"ContainerStarted","Data":"756b0c3868f46c43a332aee063ba1f81cd4fbf54ee07e954ed405fe3a7d50d94"}
Sep 11 18:13:32.269292 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:32.269255    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx" event={"ID":"0b41d347-0b15-498c-b44b-6771e8ced079","Type":"ContainerStarted","Data":"e179583e9297a8fb8c4c5e4da5ae9f223c91aa05e4641be47989c1e7ff742029"}
Sep 11 18:13:32.269561 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:32.269388    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx"
Sep 11 18:13:32.270953 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:32.270923    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g" event={"ID":"4d67ff0b-8104-45d1-8eee-49b41fb9edf9","Type":"ContainerStarted","Data":"38717ca797f2dc9e4871d997807b6bbb49a177dc65a9839a28c04f003784a809"}
Sep 11 18:13:32.271089 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:32.270976    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g" event={"ID":"4d67ff0b-8104-45d1-8eee-49b41fb9edf9","Type":"ContainerStarted","Data":"a7a02dd26df05257be1fd321da6f24901a6d12ba77dbfa220b64102866a4874d"}
Sep 11 18:13:32.271089 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:32.271004    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g"
Sep 11 18:13:32.289522 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:32.289469    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx" podStartSLOduration=1.289452247 podStartE2EDuration="1.289452247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:13:32.287394888 +0000 UTC m=+1717.732495461" watchObservedRunningTime="2026-09-11 18:13:32.289452247 +0000 UTC m=+1717.734552873"
Sep 11 18:13:32.306714 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:32.306653    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g" podStartSLOduration=1.306632202 podStartE2EDuration="1.306632202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:13:32.305955339 +0000 UTC m=+1717.751055911" watchObservedRunningTime="2026-09-11 18:13:32.306632202 +0000 UTC m=+1717.751732773"
Sep 11 18:13:36.441331 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:36.441294    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:13:43.280021 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:43.279989    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx"
Sep 11 18:13:43.280542 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:13:43.280068    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g"
Sep 11 18:14:08.910810 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:08.910770    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g"]
Sep 11 18:14:08.911419 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:08.911045    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g" podUID="4d67ff0b-8104-45d1-8eee-49b41fb9edf9" containerName="payload-processing" containerID="cri-o://38717ca797f2dc9e4871d997807b6bbb49a177dc65a9839a28c04f003784a809" gracePeriod=30
Sep 11 18:14:08.916515 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:08.916483    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx"]
Sep 11 18:14:08.916799 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:08.916750    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx" podUID="0b41d347-0b15-498c-b44b-6771e8ced079" containerName="payload-pre-processing" containerID="cri-o://756b0c3868f46c43a332aee063ba1f81cd4fbf54ee07e954ed405fe3a7d50d94" gracePeriod=30
Sep 11 18:14:09.204410 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.204381    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g"
Sep 11 18:14:09.207382 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.207338    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx"
Sep 11 18:14:09.277937 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.277896    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d99xp\" (UniqueName: \"kubernetes.io/projected/0b41d347-0b15-498c-b44b-6771e8ced079-kube-api-access-d99xp\") pod \"0b41d347-0b15-498c-b44b-6771e8ced079\" (UID: \"0b41d347-0b15-498c-b44b-6771e8ced079\") "
Sep 11 18:14:09.277937 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.277941    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b41d347-0b15-498c-b44b-6771e8ced079-plugins-config-volume\") pod \"0b41d347-0b15-498c-b44b-6771e8ced079\" (UID: \"0b41d347-0b15-498c-b44b-6771e8ced079\") "
Sep 11 18:14:09.278152 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.277969    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ztdp\" (UniqueName: \"kubernetes.io/projected/4d67ff0b-8104-45d1-8eee-49b41fb9edf9-kube-api-access-7ztdp\") pod \"4d67ff0b-8104-45d1-8eee-49b41fb9edf9\" (UID: \"4d67ff0b-8104-45d1-8eee-49b41fb9edf9\") "
Sep 11 18:14:09.278152 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.278014    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d67ff0b-8104-45d1-8eee-49b41fb9edf9-plugins-config-volume\") pod \"4d67ff0b-8104-45d1-8eee-49b41fb9edf9\" (UID: \"4d67ff0b-8104-45d1-8eee-49b41fb9edf9\") "
Sep 11 18:14:09.278477 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.278436    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b41d347-0b15-498c-b44b-6771e8ced079-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0b41d347-0b15-498c-b44b-6771e8ced079" (UID: "0b41d347-0b15-498c-b44b-6771e8ced079"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:14:09.278477 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.278460    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d67ff0b-8104-45d1-8eee-49b41fb9edf9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4d67ff0b-8104-45d1-8eee-49b41fb9edf9" (UID: "4d67ff0b-8104-45d1-8eee-49b41fb9edf9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:14:09.279985 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.279962    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b41d347-0b15-498c-b44b-6771e8ced079-kube-api-access-d99xp" (OuterVolumeSpecName: "kube-api-access-d99xp") pod "0b41d347-0b15-498c-b44b-6771e8ced079" (UID: "0b41d347-0b15-498c-b44b-6771e8ced079"). InnerVolumeSpecName "kube-api-access-d99xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:14:09.280090 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.280032    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d67ff0b-8104-45d1-8eee-49b41fb9edf9-kube-api-access-7ztdp" (OuterVolumeSpecName: "kube-api-access-7ztdp") pod "4d67ff0b-8104-45d1-8eee-49b41fb9edf9" (UID: "4d67ff0b-8104-45d1-8eee-49b41fb9edf9"). InnerVolumeSpecName "kube-api-access-7ztdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:14:09.379337 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.379302    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d99xp\" (UniqueName: \"kubernetes.io/projected/0b41d347-0b15-498c-b44b-6771e8ced079-kube-api-access-d99xp\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:14:09.379337 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.379330    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b41d347-0b15-498c-b44b-6771e8ced079-plugins-config-volume\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:14:09.379337 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.379340    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ztdp\" (UniqueName: \"kubernetes.io/projected/4d67ff0b-8104-45d1-8eee-49b41fb9edf9-kube-api-access-7ztdp\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:14:09.379337 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.379363    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d67ff0b-8104-45d1-8eee-49b41fb9edf9-plugins-config-volume\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:14:09.488879 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.488773    2559 generic.go:356] "Generic (PLEG): container finished" podID="4d67ff0b-8104-45d1-8eee-49b41fb9edf9" containerID="38717ca797f2dc9e4871d997807b6bbb49a177dc65a9839a28c04f003784a809" exitCode=0
Sep 11 18:14:09.488879 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.488826    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g"
Sep 11 18:14:09.488879 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.488868    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g" event={"ID":"4d67ff0b-8104-45d1-8eee-49b41fb9edf9","Type":"ContainerDied","Data":"38717ca797f2dc9e4871d997807b6bbb49a177dc65a9839a28c04f003784a809"}
Sep 11 18:14:09.489162 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.488922    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g" event={"ID":"4d67ff0b-8104-45d1-8eee-49b41fb9edf9","Type":"ContainerDied","Data":"a7a02dd26df05257be1fd321da6f24901a6d12ba77dbfa220b64102866a4874d"}
Sep 11 18:14:09.489162 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.488945    2559 scope.go:117] "RemoveContainer" containerID="38717ca797f2dc9e4871d997807b6bbb49a177dc65a9839a28c04f003784a809"
Sep 11 18:14:09.490529 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.490506    2559 generic.go:356] "Generic (PLEG): container finished" podID="0b41d347-0b15-498c-b44b-6771e8ced079" containerID="756b0c3868f46c43a332aee063ba1f81cd4fbf54ee07e954ed405fe3a7d50d94" exitCode=0
Sep 11 18:14:09.490693 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.490552    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx"
Sep 11 18:14:09.490693 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.490577    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx" event={"ID":"0b41d347-0b15-498c-b44b-6771e8ced079","Type":"ContainerDied","Data":"756b0c3868f46c43a332aee063ba1f81cd4fbf54ee07e954ed405fe3a7d50d94"}
Sep 11 18:14:09.490693 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.490614    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx" event={"ID":"0b41d347-0b15-498c-b44b-6771e8ced079","Type":"ContainerDied","Data":"e179583e9297a8fb8c4c5e4da5ae9f223c91aa05e4641be47989c1e7ff742029"}
Sep 11 18:14:09.499001 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.498981    2559 scope.go:117] "RemoveContainer" containerID="38717ca797f2dc9e4871d997807b6bbb49a177dc65a9839a28c04f003784a809"
Sep 11 18:14:09.499297 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:14:09.499277    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38717ca797f2dc9e4871d997807b6bbb49a177dc65a9839a28c04f003784a809\": container with ID starting with 38717ca797f2dc9e4871d997807b6bbb49a177dc65a9839a28c04f003784a809 not found: ID does not exist" containerID="38717ca797f2dc9e4871d997807b6bbb49a177dc65a9839a28c04f003784a809"
Sep 11 18:14:09.499377 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.499305    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38717ca797f2dc9e4871d997807b6bbb49a177dc65a9839a28c04f003784a809"} err="failed to get container status \"38717ca797f2dc9e4871d997807b6bbb49a177dc65a9839a28c04f003784a809\": rpc error: code = NotFound desc = could not find container \"38717ca797f2dc9e4871d997807b6bbb49a177dc65a9839a28c04f003784a809\": container with ID starting with 38717ca797f2dc9e4871d997807b6bbb49a177dc65a9839a28c04f003784a809 not found: ID does not exist"
Sep 11 18:14:09.499377 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.499322    2559 scope.go:117] "RemoveContainer" containerID="756b0c3868f46c43a332aee063ba1f81cd4fbf54ee07e954ed405fe3a7d50d94"
Sep 11 18:14:09.507728 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.507709    2559 scope.go:117] "RemoveContainer" containerID="756b0c3868f46c43a332aee063ba1f81cd4fbf54ee07e954ed405fe3a7d50d94"
Sep 11 18:14:09.507971 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:14:09.507954    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"756b0c3868f46c43a332aee063ba1f81cd4fbf54ee07e954ed405fe3a7d50d94\": container with ID starting with 756b0c3868f46c43a332aee063ba1f81cd4fbf54ee07e954ed405fe3a7d50d94 not found: ID does not exist" containerID="756b0c3868f46c43a332aee063ba1f81cd4fbf54ee07e954ed405fe3a7d50d94"
Sep 11 18:14:09.508017 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.507978    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"756b0c3868f46c43a332aee063ba1f81cd4fbf54ee07e954ed405fe3a7d50d94"} err="failed to get container status \"756b0c3868f46c43a332aee063ba1f81cd4fbf54ee07e954ed405fe3a7d50d94\": rpc error: code = NotFound desc = could not find container \"756b0c3868f46c43a332aee063ba1f81cd4fbf54ee07e954ed405fe3a7d50d94\": container with ID starting with 756b0c3868f46c43a332aee063ba1f81cd4fbf54ee07e954ed405fe3a7d50d94 not found: ID does not exist"
Sep 11 18:14:09.514882 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.514855    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g"]
Sep 11 18:14:09.517827 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.517806    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-4f951a-6748d68859-z946g"]
Sep 11 18:14:09.527634 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.527612    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx"]
Sep 11 18:14:09.530791 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:09.530770    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4f951a-54d87c9b58-bzhzx"]
Sep 11 18:14:11.053048 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:11.053008    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b41d347-0b15-498c-b44b-6771e8ced079" path="/var/lib/kubelet/pods/0b41d347-0b15-498c-b44b-6771e8ced079/volumes"
Sep 11 18:14:11.053590 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:11.053569    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d67ff0b-8104-45d1-8eee-49b41fb9edf9" path="/var/lib/kubelet/pods/4d67ff0b-8104-45d1-8eee-49b41fb9edf9/volumes"
Sep 11 18:14:13.431178 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:13.431142    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:14:33.030929 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:33.030892    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:14:35.471563 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:35.471531    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq"]
Sep 11 18:14:35.472064 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:35.471792    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq" podUID="42e97169-c6ca-4575-8896-3f10f4f21836" containerName="payload-pre-processing" containerID="cri-o://9e630cbceebfd0d50e9b727b3f6f561bc492028c746b2901e2377fafb40fd4fa" gracePeriod=30
Sep 11 18:14:35.640464 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:35.640431    2559 generic.go:356] "Generic (PLEG): container finished" podID="42e97169-c6ca-4575-8896-3f10f4f21836" containerID="9e630cbceebfd0d50e9b727b3f6f561bc492028c746b2901e2377fafb40fd4fa" exitCode=0
Sep 11 18:14:35.640634 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:35.640507    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq" event={"ID":"42e97169-c6ca-4575-8896-3f10f4f21836","Type":"ContainerDied","Data":"9e630cbceebfd0d50e9b727b3f6f561bc492028c746b2901e2377fafb40fd4fa"}
Sep 11 18:14:35.729172 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:35.729112    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq"
Sep 11 18:14:35.844724 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:35.844687    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e97169-c6ca-4575-8896-3f10f4f21836-plugins-config-volume\") pod \"42e97169-c6ca-4575-8896-3f10f4f21836\" (UID: \"42e97169-c6ca-4575-8896-3f10f4f21836\") "
Sep 11 18:14:35.844918 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:35.844782    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65t9s\" (UniqueName: \"kubernetes.io/projected/42e97169-c6ca-4575-8896-3f10f4f21836-kube-api-access-65t9s\") pod \"42e97169-c6ca-4575-8896-3f10f4f21836\" (UID: \"42e97169-c6ca-4575-8896-3f10f4f21836\") "
Sep 11 18:14:35.845087 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:35.845060    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42e97169-c6ca-4575-8896-3f10f4f21836-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42e97169-c6ca-4575-8896-3f10f4f21836" (UID: "42e97169-c6ca-4575-8896-3f10f4f21836"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:14:35.846892 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:35.846869    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e97169-c6ca-4575-8896-3f10f4f21836-kube-api-access-65t9s" (OuterVolumeSpecName: "kube-api-access-65t9s") pod "42e97169-c6ca-4575-8896-3f10f4f21836" (UID: "42e97169-c6ca-4575-8896-3f10f4f21836"). InnerVolumeSpecName "kube-api-access-65t9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:14:35.945980 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:35.945942    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e97169-c6ca-4575-8896-3f10f4f21836-plugins-config-volume\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:14:35.945980 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:35.945975    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65t9s\" (UniqueName: \"kubernetes.io/projected/42e97169-c6ca-4575-8896-3f10f4f21836-kube-api-access-65t9s\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:14:36.647280 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:36.647247    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq"
Sep 11 18:14:36.647745 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:36.647281    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq" event={"ID":"42e97169-c6ca-4575-8896-3f10f4f21836","Type":"ContainerDied","Data":"9aa0316d973031718369e6bbed908a0ae27f67bf66ae236a328e6618fbb6b20c"}
Sep 11 18:14:36.647745 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:36.647341    2559 scope.go:117] "RemoveContainer" containerID="9e630cbceebfd0d50e9b727b3f6f561bc492028c746b2901e2377fafb40fd4fa"
Sep 11 18:14:36.668953 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:36.668923    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq"]
Sep 11 18:14:36.672972 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:36.672947    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6ad962-58847bcbc8-cdtwq"]
Sep 11 18:14:37.054717 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:37.054636    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42e97169-c6ca-4575-8896-3f10f4f21836" path="/var/lib/kubelet/pods/42e97169-c6ca-4575-8896-3f10f4f21836/volumes"
Sep 11 18:14:41.026912 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:41.026879    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"]
Sep 11 18:14:45.725977 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:45.725937    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:14:56.046875 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.046811    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl" podUID="998837d8-d744-40bf-98f6-eed6a4ccec08" containerName="main" containerID="cri-o://01d2c91c74fe50b52969a33cc43badd6b0ed7823bc091a8dd7ce17b651a769f8" gracePeriod=45
Sep 11 18:14:56.124058 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.124006    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl" podUID="998837d8-d744-40bf-98f6-eed6a4ccec08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": dial tcp 10.132.0.87:8000: connect: connection refused"
Sep 11 18:14:56.298836 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.298775    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:14:56.343390 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.343333    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/998837d8-d744-40bf-98f6-eed6a4ccec08-tmp-dir\") pod \"998837d8-d744-40bf-98f6-eed6a4ccec08\" (UID: \"998837d8-d744-40bf-98f6-eed6a4ccec08\") "
Sep 11 18:14:56.343571 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.343461    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/998837d8-d744-40bf-98f6-eed6a4ccec08-tls-certs\") pod \"998837d8-d744-40bf-98f6-eed6a4ccec08\" (UID: \"998837d8-d744-40bf-98f6-eed6a4ccec08\") "
Sep 11 18:14:56.343571 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.343490    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22d8j\" (UniqueName: \"kubernetes.io/projected/998837d8-d744-40bf-98f6-eed6a4ccec08-kube-api-access-22d8j\") pod \"998837d8-d744-40bf-98f6-eed6a4ccec08\" (UID: \"998837d8-d744-40bf-98f6-eed6a4ccec08\") "
Sep 11 18:14:56.343571 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.343538    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/998837d8-d744-40bf-98f6-eed6a4ccec08-home\") pod \"998837d8-d744-40bf-98f6-eed6a4ccec08\" (UID: \"998837d8-d744-40bf-98f6-eed6a4ccec08\") "
Sep 11 18:14:56.343749 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.343576    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/998837d8-d744-40bf-98f6-eed6a4ccec08-dshm\") pod \"998837d8-d744-40bf-98f6-eed6a4ccec08\" (UID: \"998837d8-d744-40bf-98f6-eed6a4ccec08\") "
Sep 11 18:14:56.343749 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.343609    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998837d8-d744-40bf-98f6-eed6a4ccec08-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "998837d8-d744-40bf-98f6-eed6a4ccec08" (UID: "998837d8-d744-40bf-98f6-eed6a4ccec08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:14:56.343749 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.343644    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/998837d8-d744-40bf-98f6-eed6a4ccec08-model-cache\") pod \"998837d8-d744-40bf-98f6-eed6a4ccec08\" (UID: \"998837d8-d744-40bf-98f6-eed6a4ccec08\") "
Sep 11 18:14:56.343973 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.343944    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998837d8-d744-40bf-98f6-eed6a4ccec08-home" (OuterVolumeSpecName: "home") pod "998837d8-d744-40bf-98f6-eed6a4ccec08" (UID: "998837d8-d744-40bf-98f6-eed6a4ccec08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:14:56.344075 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.343948    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998837d8-d744-40bf-98f6-eed6a4ccec08-model-cache" (OuterVolumeSpecName: "model-cache") pod "998837d8-d744-40bf-98f6-eed6a4ccec08" (UID: "998837d8-d744-40bf-98f6-eed6a4ccec08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:14:56.344284 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.344257    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/998837d8-d744-40bf-98f6-eed6a4ccec08-home\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:14:56.344456 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.344288    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/998837d8-d744-40bf-98f6-eed6a4ccec08-model-cache\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:14:56.344456 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.344303    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/998837d8-d744-40bf-98f6-eed6a4ccec08-tmp-dir\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:14:56.345844 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.345809    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998837d8-d744-40bf-98f6-eed6a4ccec08-dshm" (OuterVolumeSpecName: "dshm") pod "998837d8-d744-40bf-98f6-eed6a4ccec08" (UID: "998837d8-d744-40bf-98f6-eed6a4ccec08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:14:56.345966 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.345939    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998837d8-d744-40bf-98f6-eed6a4ccec08-kube-api-access-22d8j" (OuterVolumeSpecName: "kube-api-access-22d8j") pod "998837d8-d744-40bf-98f6-eed6a4ccec08" (UID: "998837d8-d744-40bf-98f6-eed6a4ccec08"). InnerVolumeSpecName "kube-api-access-22d8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:14:56.346248 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.346219    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/998837d8-d744-40bf-98f6-eed6a4ccec08-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "998837d8-d744-40bf-98f6-eed6a4ccec08" (UID: "998837d8-d744-40bf-98f6-eed6a4ccec08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:14:56.445124 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.445084    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/998837d8-d744-40bf-98f6-eed6a4ccec08-tls-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:14:56.445124 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.445119    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22d8j\" (UniqueName: \"kubernetes.io/projected/998837d8-d744-40bf-98f6-eed6a4ccec08-kube-api-access-22d8j\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:14:56.445124 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.445128    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/998837d8-d744-40bf-98f6-eed6a4ccec08-dshm\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:14:56.755824 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.755790    2559 generic.go:356] "Generic (PLEG): container finished" podID="998837d8-d744-40bf-98f6-eed6a4ccec08" containerID="01d2c91c74fe50b52969a33cc43badd6b0ed7823bc091a8dd7ce17b651a769f8" exitCode=0
Sep 11 18:14:56.756009 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.755837    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"
Sep 11 18:14:56.756009 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.755859    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl" event={"ID":"998837d8-d744-40bf-98f6-eed6a4ccec08","Type":"ContainerDied","Data":"01d2c91c74fe50b52969a33cc43badd6b0ed7823bc091a8dd7ce17b651a769f8"}
Sep 11 18:14:56.756009 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.755890    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl" event={"ID":"998837d8-d744-40bf-98f6-eed6a4ccec08","Type":"ContainerDied","Data":"6e7a3894e76ecc918726a6ea1815a3864637438822aaae88d4718ac352c58469"}
Sep 11 18:14:56.756009 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.755906    2559 scope.go:117] "RemoveContainer" containerID="01d2c91c74fe50b52969a33cc43badd6b0ed7823bc091a8dd7ce17b651a769f8"
Sep 11 18:14:56.765340 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.765321    2559 scope.go:117] "RemoveContainer" containerID="01d2c91c74fe50b52969a33cc43badd6b0ed7823bc091a8dd7ce17b651a769f8"
Sep 11 18:14:56.765691 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:14:56.765671    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01d2c91c74fe50b52969a33cc43badd6b0ed7823bc091a8dd7ce17b651a769f8\": container with ID starting with 01d2c91c74fe50b52969a33cc43badd6b0ed7823bc091a8dd7ce17b651a769f8 not found: ID does not exist" containerID="01d2c91c74fe50b52969a33cc43badd6b0ed7823bc091a8dd7ce17b651a769f8"
Sep 11 18:14:56.765732 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.765701    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d2c91c74fe50b52969a33cc43badd6b0ed7823bc091a8dd7ce17b651a769f8"} err="failed to get container status \"01d2c91c74fe50b52969a33cc43badd6b0ed7823bc091a8dd7ce17b651a769f8\": rpc error: code = NotFound desc = could not find container \"01d2c91c74fe50b52969a33cc43badd6b0ed7823bc091a8dd7ce17b651a769f8\": container with ID starting with 01d2c91c74fe50b52969a33cc43badd6b0ed7823bc091a8dd7ce17b651a769f8 not found: ID does not exist"
Sep 11 18:14:56.778397 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.778371    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"]
Sep 11 18:14:56.782245 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:56.782221    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-6ad962/ipp-route-6ad962-kserve-679554c455-s95pl"]
Sep 11 18:14:57.054009 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:14:57.053915    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="998837d8-d744-40bf-98f6-eed6a4ccec08" path="/var/lib/kubelet/pods/998837d8-d744-40bf-98f6-eed6a4ccec08/volumes"
Sep 11 18:15:35.267334 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.267296    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn"]
Sep 11 18:15:35.267888 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.267556    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn" podUID="c628c036-8daf-4e06-ac35-e6506a639f33" containerName="maas-api" containerID="cri-o://fd1b8fcf842ec46d7c8905e5a55795b27dc0e58c5655f2ace56cd43f94e425ec" gracePeriod=30
Sep 11 18:15:35.330241 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.330208    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95"]
Sep 11 18:15:35.330460 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.330432    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95" podUID="4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb" containerName="payload-processing" containerID="cri-o://fff8ccbb1017a38db8a1b90c129ac745a1dc000065d65f53279780e1dc00d437" gracePeriod=30
Sep 11 18:15:35.577214 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.577183    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn"
Sep 11 18:15:35.612866 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.612830    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95"
Sep 11 18:15:35.615761 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.615742    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c628c036-8daf-4e06-ac35-e6506a639f33-openshift-service-ca\") pod \"c628c036-8daf-4e06-ac35-e6506a639f33\" (UID: \"c628c036-8daf-4e06-ac35-e6506a639f33\") "
Sep 11 18:15:35.615858 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.615841    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrps8\" (UniqueName: \"kubernetes.io/projected/c628c036-8daf-4e06-ac35-e6506a639f33-kube-api-access-vrps8\") pod \"c628c036-8daf-4e06-ac35-e6506a639f33\" (UID: \"c628c036-8daf-4e06-ac35-e6506a639f33\") "
Sep 11 18:15:35.615949 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.615934    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c628c036-8daf-4e06-ac35-e6506a639f33-maas-api-tls\") pod \"c628c036-8daf-4e06-ac35-e6506a639f33\" (UID: \"c628c036-8daf-4e06-ac35-e6506a639f33\") "
Sep 11 18:15:35.616012 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.615978    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c628c036-8daf-4e06-ac35-e6506a639f33-metrics-certs\") pod \"c628c036-8daf-4e06-ac35-e6506a639f33\" (UID: \"c628c036-8daf-4e06-ac35-e6506a639f33\") "
Sep 11 18:15:35.616155 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.616133    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c628c036-8daf-4e06-ac35-e6506a639f33-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c628c036-8daf-4e06-ac35-e6506a639f33" (UID: "c628c036-8daf-4e06-ac35-e6506a639f33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:15:35.616320 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.616290    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c628c036-8daf-4e06-ac35-e6506a639f33-openshift-service-ca\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:15:35.617970 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.617942    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c628c036-8daf-4e06-ac35-e6506a639f33-kube-api-access-vrps8" (OuterVolumeSpecName: "kube-api-access-vrps8") pod "c628c036-8daf-4e06-ac35-e6506a639f33" (UID: "c628c036-8daf-4e06-ac35-e6506a639f33"). InnerVolumeSpecName "kube-api-access-vrps8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:15:35.618495 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.618466    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c628c036-8daf-4e06-ac35-e6506a639f33-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c628c036-8daf-4e06-ac35-e6506a639f33" (UID: "c628c036-8daf-4e06-ac35-e6506a639f33"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:15:35.618568 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.618492    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c628c036-8daf-4e06-ac35-e6506a639f33-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c628c036-8daf-4e06-ac35-e6506a639f33" (UID: "c628c036-8daf-4e06-ac35-e6506a639f33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:15:35.717272 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.717232    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gn8xb\" (UniqueName: \"kubernetes.io/projected/4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb-kube-api-access-gn8xb\") pod \"4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb\" (UID: \"4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb\") "
Sep 11 18:15:35.717470 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.717439    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb-plugins-config-volume\") pod \"4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb\" (UID: \"4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb\") "
Sep 11 18:15:35.717771 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.717754    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrps8\" (UniqueName: \"kubernetes.io/projected/c628c036-8daf-4e06-ac35-e6506a639f33-kube-api-access-vrps8\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:15:35.717810 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.717774    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c628c036-8daf-4e06-ac35-e6506a639f33-maas-api-tls\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:15:35.717810 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.717784    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c628c036-8daf-4e06-ac35-e6506a639f33-metrics-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:15:35.717810 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.717782    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb" (UID: "4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:15:35.719413 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.719380    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb-kube-api-access-gn8xb" (OuterVolumeSpecName: "kube-api-access-gn8xb") pod "4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb" (UID: "4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb"). InnerVolumeSpecName "kube-api-access-gn8xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:15:35.818617 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.818520    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb-plugins-config-volume\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:15:35.818617 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.818552    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn8xb\" (UniqueName: \"kubernetes.io/projected/4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb-kube-api-access-gn8xb\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:15:35.975249 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.975215    2559 generic.go:356] "Generic (PLEG): container finished" podID="c628c036-8daf-4e06-ac35-e6506a639f33" containerID="fd1b8fcf842ec46d7c8905e5a55795b27dc0e58c5655f2ace56cd43f94e425ec" exitCode=0
Sep 11 18:15:35.975461 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.975263    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn"
Sep 11 18:15:35.975461 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.975306    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn" event={"ID":"c628c036-8daf-4e06-ac35-e6506a639f33","Type":"ContainerDied","Data":"fd1b8fcf842ec46d7c8905e5a55795b27dc0e58c5655f2ace56cd43f94e425ec"}
Sep 11 18:15:35.975461 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.975387    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn" event={"ID":"c628c036-8daf-4e06-ac35-e6506a639f33","Type":"ContainerDied","Data":"eef04d3002161a55798b8dfe94037e85c4aea1fe3b900eb39b9564ddfa05228f"}
Sep 11 18:15:35.975461 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.975412    2559 scope.go:117] "RemoveContainer" containerID="fd1b8fcf842ec46d7c8905e5a55795b27dc0e58c5655f2ace56cd43f94e425ec"
Sep 11 18:15:35.976718 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.976699    2559 generic.go:356] "Generic (PLEG): container finished" podID="4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb" containerID="fff8ccbb1017a38db8a1b90c129ac745a1dc000065d65f53279780e1dc00d437" exitCode=0
Sep 11 18:15:35.976803 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.976749    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95"
Sep 11 18:15:35.976866 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.976790    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95" event={"ID":"4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb","Type":"ContainerDied","Data":"fff8ccbb1017a38db8a1b90c129ac745a1dc000065d65f53279780e1dc00d437"}
Sep 11 18:15:35.976866 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.976845    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95" event={"ID":"4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb","Type":"ContainerDied","Data":"2ec1739f785311bf3282ceb1fd037ff50bf7eff982a13391eb6c75091e67c425"}
Sep 11 18:15:35.986641 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.986621    2559 scope.go:117] "RemoveContainer" containerID="fd1b8fcf842ec46d7c8905e5a55795b27dc0e58c5655f2ace56cd43f94e425ec"
Sep 11 18:15:35.987010 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:15:35.986955    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd1b8fcf842ec46d7c8905e5a55795b27dc0e58c5655f2ace56cd43f94e425ec\": container with ID starting with fd1b8fcf842ec46d7c8905e5a55795b27dc0e58c5655f2ace56cd43f94e425ec not found: ID does not exist" containerID="fd1b8fcf842ec46d7c8905e5a55795b27dc0e58c5655f2ace56cd43f94e425ec"
Sep 11 18:15:35.987010 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.986994    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd1b8fcf842ec46d7c8905e5a55795b27dc0e58c5655f2ace56cd43f94e425ec"} err="failed to get container status \"fd1b8fcf842ec46d7c8905e5a55795b27dc0e58c5655f2ace56cd43f94e425ec\": rpc error: code = NotFound desc = could not find container \"fd1b8fcf842ec46d7c8905e5a55795b27dc0e58c5655f2ace56cd43f94e425ec\": container with ID starting with fd1b8fcf842ec46d7c8905e5a55795b27dc0e58c5655f2ace56cd43f94e425ec not found: ID does not exist"
Sep 11 18:15:35.987138 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.987020    2559 scope.go:117] "RemoveContainer" containerID="fff8ccbb1017a38db8a1b90c129ac745a1dc000065d65f53279780e1dc00d437"
Sep 11 18:15:35.995557 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.995533    2559 scope.go:117] "RemoveContainer" containerID="fff8ccbb1017a38db8a1b90c129ac745a1dc000065d65f53279780e1dc00d437"
Sep 11 18:15:35.995840 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:15:35.995819    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fff8ccbb1017a38db8a1b90c129ac745a1dc000065d65f53279780e1dc00d437\": container with ID starting with fff8ccbb1017a38db8a1b90c129ac745a1dc000065d65f53279780e1dc00d437 not found: ID does not exist" containerID="fff8ccbb1017a38db8a1b90c129ac745a1dc000065d65f53279780e1dc00d437"
Sep 11 18:15:35.995906 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:35.995852    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff8ccbb1017a38db8a1b90c129ac745a1dc000065d65f53279780e1dc00d437"} err="failed to get container status \"fff8ccbb1017a38db8a1b90c129ac745a1dc000065d65f53279780e1dc00d437\": rpc error: code = NotFound desc = could not find container \"fff8ccbb1017a38db8a1b90c129ac745a1dc000065d65f53279780e1dc00d437\": container with ID starting with fff8ccbb1017a38db8a1b90c129ac745a1dc000065d65f53279780e1dc00d437 not found: ID does not exist"
Sep 11 18:15:36.001376 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:36.001337    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn"]
Sep 11 18:15:36.007053 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:36.007027    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b54eb6-5f9dfdff7c-982cn"]
Sep 11 18:15:36.017275 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:36.017243    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95"]
Sep 11 18:15:36.020158 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:36.020132    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b54eb6-575695cb5f-wbg95"]
Sep 11 18:15:37.053715 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:37.053678    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb" path="/var/lib/kubelet/pods/4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb/volumes"
Sep 11 18:15:37.054284 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:37.054262    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c628c036-8daf-4e06-ac35-e6506a639f33" path="/var/lib/kubelet/pods/c628c036-8daf-4e06-ac35-e6506a639f33/volumes"
Sep 11 18:15:38.548538 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:38.548504    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:15:53.556874 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:53.556840    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h"]
Sep 11 18:15:53.557263 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:53.557059    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h" podUID="4cda3d66-3ee6-468f-9fd4-fab720ed56c2" containerName="maas-api" containerID="cri-o://05eeb622f61f136742f0fe213784b4dc17ad8df90accd513b658947f1175fadf" gracePeriod=30
Sep 11 18:15:53.810793 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:53.810731    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h"
Sep 11 18:15:53.897919 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:53.897886    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-654fr\" (UniqueName: \"kubernetes.io/projected/4cda3d66-3ee6-468f-9fd4-fab720ed56c2-kube-api-access-654fr\") pod \"4cda3d66-3ee6-468f-9fd4-fab720ed56c2\" (UID: \"4cda3d66-3ee6-468f-9fd4-fab720ed56c2\") "
Sep 11 18:15:53.897919 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:53.897922    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cda3d66-3ee6-468f-9fd4-fab720ed56c2-metrics-certs\") pod \"4cda3d66-3ee6-468f-9fd4-fab720ed56c2\" (UID: \"4cda3d66-3ee6-468f-9fd4-fab720ed56c2\") "
Sep 11 18:15:53.898122 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:53.897980    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cda3d66-3ee6-468f-9fd4-fab720ed56c2-maas-api-tls\") pod \"4cda3d66-3ee6-468f-9fd4-fab720ed56c2\" (UID: \"4cda3d66-3ee6-468f-9fd4-fab720ed56c2\") "
Sep 11 18:15:53.898122 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:53.897998    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cda3d66-3ee6-468f-9fd4-fab720ed56c2-openshift-service-ca\") pod \"4cda3d66-3ee6-468f-9fd4-fab720ed56c2\" (UID: \"4cda3d66-3ee6-468f-9fd4-fab720ed56c2\") "
Sep 11 18:15:53.898482 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:53.898448    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cda3d66-3ee6-468f-9fd4-fab720ed56c2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4cda3d66-3ee6-468f-9fd4-fab720ed56c2" (UID: "4cda3d66-3ee6-468f-9fd4-fab720ed56c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:15:53.899958 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:53.899937    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cda3d66-3ee6-468f-9fd4-fab720ed56c2-kube-api-access-654fr" (OuterVolumeSpecName: "kube-api-access-654fr") pod "4cda3d66-3ee6-468f-9fd4-fab720ed56c2" (UID: "4cda3d66-3ee6-468f-9fd4-fab720ed56c2"). InnerVolumeSpecName "kube-api-access-654fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:15:53.900094 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:53.900081    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cda3d66-3ee6-468f-9fd4-fab720ed56c2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4cda3d66-3ee6-468f-9fd4-fab720ed56c2" (UID: "4cda3d66-3ee6-468f-9fd4-fab720ed56c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:15:53.900160 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:53.900145    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cda3d66-3ee6-468f-9fd4-fab720ed56c2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4cda3d66-3ee6-468f-9fd4-fab720ed56c2" (UID: "4cda3d66-3ee6-468f-9fd4-fab720ed56c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:15:53.999091 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:53.999053    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-654fr\" (UniqueName: \"kubernetes.io/projected/4cda3d66-3ee6-468f-9fd4-fab720ed56c2-kube-api-access-654fr\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:15:53.999091 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:53.999083    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cda3d66-3ee6-468f-9fd4-fab720ed56c2-metrics-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:15:53.999091 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:53.999092    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cda3d66-3ee6-468f-9fd4-fab720ed56c2-maas-api-tls\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:15:53.999091 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:53.999100    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cda3d66-3ee6-468f-9fd4-fab720ed56c2-openshift-service-ca\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:15:54.074157 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:54.074069    2559 generic.go:356] "Generic (PLEG): container finished" podID="4cda3d66-3ee6-468f-9fd4-fab720ed56c2" containerID="05eeb622f61f136742f0fe213784b4dc17ad8df90accd513b658947f1175fadf" exitCode=0
Sep 11 18:15:54.074157 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:54.074107    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h"
Sep 11 18:15:54.074157 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:54.074147    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h" event={"ID":"4cda3d66-3ee6-468f-9fd4-fab720ed56c2","Type":"ContainerDied","Data":"05eeb622f61f136742f0fe213784b4dc17ad8df90accd513b658947f1175fadf"}
Sep 11 18:15:54.074406 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:54.074205    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h" event={"ID":"4cda3d66-3ee6-468f-9fd4-fab720ed56c2","Type":"ContainerDied","Data":"8b16e4db41e006480535443e5062b92a42a7f4a6c37fd6f32e9e2951762932d8"}
Sep 11 18:15:54.074406 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:54.074239    2559 scope.go:117] "RemoveContainer" containerID="05eeb622f61f136742f0fe213784b4dc17ad8df90accd513b658947f1175fadf"
Sep 11 18:15:54.093362 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:54.093330    2559 scope.go:117] "RemoveContainer" containerID="05eeb622f61f136742f0fe213784b4dc17ad8df90accd513b658947f1175fadf"
Sep 11 18:15:54.093675 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:15:54.093657    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05eeb622f61f136742f0fe213784b4dc17ad8df90accd513b658947f1175fadf\": container with ID starting with 05eeb622f61f136742f0fe213784b4dc17ad8df90accd513b658947f1175fadf not found: ID does not exist" containerID="05eeb622f61f136742f0fe213784b4dc17ad8df90accd513b658947f1175fadf"
Sep 11 18:15:54.093738 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:54.093683    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05eeb622f61f136742f0fe213784b4dc17ad8df90accd513b658947f1175fadf"} err="failed to get container status \"05eeb622f61f136742f0fe213784b4dc17ad8df90accd513b658947f1175fadf\": rpc error: code = NotFound desc = could not find container \"05eeb622f61f136742f0fe213784b4dc17ad8df90accd513b658947f1175fadf\": container with ID starting with 05eeb622f61f136742f0fe213784b4dc17ad8df90accd513b658947f1175fadf not found: ID does not exist"
Sep 11 18:15:54.104725 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:54.104701    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h"]
Sep 11 18:15:54.108597 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:54.108575    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-386935-59bfbdbdf7-gt82h"]
Sep 11 18:15:55.053872 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:55.053828    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cda3d66-3ee6-468f-9fd4-fab720ed56c2" path="/var/lib/kubelet/pods/4cda3d66-3ee6-468f-9fd4-fab720ed56c2/volumes"
Sep 11 18:15:56.529457 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:15:56.529422    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:16:11.446774 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:11.446721    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"]
Sep 11 18:16:11.447374 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:11.446938    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m" podUID="1768d910-3783-43aa-8b6c-1857bdc4b3c8" containerName="maas-api" containerID="cri-o://00ec66916f062fa3ba9cdc3fa04613a16a444a20b059cc89d1965936b388e450" gracePeriod=30
Sep 11 18:16:11.706814 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:11.706744    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"
Sep 11 18:16:11.758812 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:11.758776    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srgnl\" (UniqueName: \"kubernetes.io/projected/1768d910-3783-43aa-8b6c-1857bdc4b3c8-kube-api-access-srgnl\") pod \"1768d910-3783-43aa-8b6c-1857bdc4b3c8\" (UID: \"1768d910-3783-43aa-8b6c-1857bdc4b3c8\") "
Sep 11 18:16:11.759000 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:11.758832    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1768d910-3783-43aa-8b6c-1857bdc4b3c8-metrics-certs\") pod \"1768d910-3783-43aa-8b6c-1857bdc4b3c8\" (UID: \"1768d910-3783-43aa-8b6c-1857bdc4b3c8\") "
Sep 11 18:16:11.759000 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:11.758884    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1768d910-3783-43aa-8b6c-1857bdc4b3c8-openshift-service-ca\") pod \"1768d910-3783-43aa-8b6c-1857bdc4b3c8\" (UID: \"1768d910-3783-43aa-8b6c-1857bdc4b3c8\") "
Sep 11 18:16:11.759000 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:11.758980    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1768d910-3783-43aa-8b6c-1857bdc4b3c8-maas-api-tls\") pod \"1768d910-3783-43aa-8b6c-1857bdc4b3c8\" (UID: \"1768d910-3783-43aa-8b6c-1857bdc4b3c8\") "
Sep 11 18:16:11.759385 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:11.759337    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1768d910-3783-43aa-8b6c-1857bdc4b3c8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1768d910-3783-43aa-8b6c-1857bdc4b3c8" (UID: "1768d910-3783-43aa-8b6c-1857bdc4b3c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:16:11.760966 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:11.760941    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1768d910-3783-43aa-8b6c-1857bdc4b3c8-kube-api-access-srgnl" (OuterVolumeSpecName: "kube-api-access-srgnl") pod "1768d910-3783-43aa-8b6c-1857bdc4b3c8" (UID: "1768d910-3783-43aa-8b6c-1857bdc4b3c8"). InnerVolumeSpecName "kube-api-access-srgnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:16:11.761084 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:11.761013    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1768d910-3783-43aa-8b6c-1857bdc4b3c8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1768d910-3783-43aa-8b6c-1857bdc4b3c8" (UID: "1768d910-3783-43aa-8b6c-1857bdc4b3c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:16:11.761132 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:11.761092    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1768d910-3783-43aa-8b6c-1857bdc4b3c8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1768d910-3783-43aa-8b6c-1857bdc4b3c8" (UID: "1768d910-3783-43aa-8b6c-1857bdc4b3c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:16:11.860550 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:11.860496    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1768d910-3783-43aa-8b6c-1857bdc4b3c8-maas-api-tls\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:16:11.860550 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:11.860544    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srgnl\" (UniqueName: \"kubernetes.io/projected/1768d910-3783-43aa-8b6c-1857bdc4b3c8-kube-api-access-srgnl\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:16:11.860550 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:11.860555    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1768d910-3783-43aa-8b6c-1857bdc4b3c8-metrics-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:16:11.860550 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:11.860566    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1768d910-3783-43aa-8b6c-1857bdc4b3c8-openshift-service-ca\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:16:12.171540 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:12.171499    2559 generic.go:356] "Generic (PLEG): container finished" podID="1768d910-3783-43aa-8b6c-1857bdc4b3c8" containerID="00ec66916f062fa3ba9cdc3fa04613a16a444a20b059cc89d1965936b388e450" exitCode=0
Sep 11 18:16:12.171755 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:12.171542    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"
Sep 11 18:16:12.171755 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:12.171591    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m" event={"ID":"1768d910-3783-43aa-8b6c-1857bdc4b3c8","Type":"ContainerDied","Data":"00ec66916f062fa3ba9cdc3fa04613a16a444a20b059cc89d1965936b388e450"}
Sep 11 18:16:12.171755 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:12.171649    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m" event={"ID":"1768d910-3783-43aa-8b6c-1857bdc4b3c8","Type":"ContainerDied","Data":"1a06fc94311d91f40e965ebfee4236e5fc557fe512f129942f7252ee50446517"}
Sep 11 18:16:12.171755 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:12.171673    2559 scope.go:117] "RemoveContainer" containerID="00ec66916f062fa3ba9cdc3fa04613a16a444a20b059cc89d1965936b388e450"
Sep 11 18:16:12.181966 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:12.181945    2559 scope.go:117] "RemoveContainer" containerID="00ec66916f062fa3ba9cdc3fa04613a16a444a20b059cc89d1965936b388e450"
Sep 11 18:16:12.182315 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:16:12.182294    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00ec66916f062fa3ba9cdc3fa04613a16a444a20b059cc89d1965936b388e450\": container with ID starting with 00ec66916f062fa3ba9cdc3fa04613a16a444a20b059cc89d1965936b388e450 not found: ID does not exist" containerID="00ec66916f062fa3ba9cdc3fa04613a16a444a20b059cc89d1965936b388e450"
Sep 11 18:16:12.182432 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:12.182321    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00ec66916f062fa3ba9cdc3fa04613a16a444a20b059cc89d1965936b388e450"} err="failed to get container status \"00ec66916f062fa3ba9cdc3fa04613a16a444a20b059cc89d1965936b388e450\": rpc error: code = NotFound desc = could not find container \"00ec66916f062fa3ba9cdc3fa04613a16a444a20b059cc89d1965936b388e450\": container with ID starting with 00ec66916f062fa3ba9cdc3fa04613a16a444a20b059cc89d1965936b388e450 not found: ID does not exist"
Sep 11 18:16:12.195309 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:12.195269    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"]
Sep 11 18:16:12.201377 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:12.201330    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3ebea5-668d97745c-vc42m"]
Sep 11 18:16:13.052910 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:13.052871    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1768d910-3783-43aa-8b6c-1857bdc4b3c8" path="/var/lib/kubelet/pods/1768d910-3783-43aa-8b6c-1857bdc4b3c8/volumes"
Sep 11 18:16:14.029294 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:14.029252    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:16:22.824428 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:16:22.824329    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:17:44.929819 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:17:44.929785    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:18:10.664428 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:18:10.664385    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:18:18.331540 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:18:18.331509    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:18:25.224690 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:18:25.224652    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:18:30.426253 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:18:30.426209    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:18:45.841264 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:18:45.841229    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:18:53.839897 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:18:53.839856    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:19:01.034837 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:19:01.034797    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:20:26.323420 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:26.323377    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:20:34.829200 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:34.829161    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:20:55.453292 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453218    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"]
Sep 11 18:20:55.453681 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453623    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42e97169-c6ca-4575-8896-3f10f4f21836" containerName="payload-pre-processing"
Sep 11 18:20:55.453681 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453637    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="42e97169-c6ca-4575-8896-3f10f4f21836" containerName="payload-pre-processing"
Sep 11 18:20:55.453681 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453647    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d67ff0b-8104-45d1-8eee-49b41fb9edf9" containerName="payload-processing"
Sep 11 18:20:55.453681 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453653    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d67ff0b-8104-45d1-8eee-49b41fb9edf9" containerName="payload-processing"
Sep 11 18:20:55.453681 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453662    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="998837d8-d744-40bf-98f6-eed6a4ccec08" containerName="main"
Sep 11 18:20:55.453681 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453668    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="998837d8-d744-40bf-98f6-eed6a4ccec08" containerName="main"
Sep 11 18:20:55.453681 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453677    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c628c036-8daf-4e06-ac35-e6506a639f33" containerName="maas-api"
Sep 11 18:20:55.453681 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453683    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c628c036-8daf-4e06-ac35-e6506a639f33" containerName="maas-api"
Sep 11 18:20:55.453987 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453693    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb" containerName="payload-processing"
Sep 11 18:20:55.453987 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453698    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb" containerName="payload-processing"
Sep 11 18:20:55.453987 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453704    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b41d347-0b15-498c-b44b-6771e8ced079" containerName="payload-pre-processing"
Sep 11 18:20:55.453987 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453710    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b41d347-0b15-498c-b44b-6771e8ced079" containerName="payload-pre-processing"
Sep 11 18:20:55.453987 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453720    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cda3d66-3ee6-468f-9fd4-fab720ed56c2" containerName="maas-api"
Sep 11 18:20:55.453987 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453725    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cda3d66-3ee6-468f-9fd4-fab720ed56c2" containerName="maas-api"
Sep 11 18:20:55.453987 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453735    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1768d910-3783-43aa-8b6c-1857bdc4b3c8" containerName="maas-api"
Sep 11 18:20:55.453987 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453740    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1768d910-3783-43aa-8b6c-1857bdc4b3c8" containerName="maas-api"
Sep 11 18:20:55.453987 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453792    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="42e97169-c6ca-4575-8896-3f10f4f21836" containerName="payload-pre-processing"
Sep 11 18:20:55.453987 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453800    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d67ff0b-8104-45d1-8eee-49b41fb9edf9" containerName="payload-processing"
Sep 11 18:20:55.453987 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453807    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1768d910-3783-43aa-8b6c-1857bdc4b3c8" containerName="maas-api"
Sep 11 18:20:55.453987 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453813    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bd64dd4-583b-4f3a-8bfa-3c7f94e777eb" containerName="payload-processing"
Sep 11 18:20:55.453987 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453818    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="998837d8-d744-40bf-98f6-eed6a4ccec08" containerName="main"
Sep 11 18:20:55.453987 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453823    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c628c036-8daf-4e06-ac35-e6506a639f33" containerName="maas-api"
Sep 11 18:20:55.453987 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453831    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cda3d66-3ee6-468f-9fd4-fab720ed56c2" containerName="maas-api"
Sep 11 18:20:55.453987 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.453836    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b41d347-0b15-498c-b44b-6771e8ced079" containerName="payload-pre-processing"
Sep 11 18:20:55.456664 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.456648    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:20:55.459421 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.459388    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-b14014df-kserve-self-signed-certs\""
Sep 11 18:20:55.466504 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.466479    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"]
Sep 11 18:20:55.482129 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.482097    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06890491-d0ee-4bc4-9e89-d7a921ed94de-tls-certs\") pod \"e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb\" (UID: \"06890491-d0ee-4bc4-9e89-d7a921ed94de\") " pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:20:55.482270 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.482163    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06890491-d0ee-4bc4-9e89-d7a921ed94de-dshm\") pod \"e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb\" (UID: \"06890491-d0ee-4bc4-9e89-d7a921ed94de\") " pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:20:55.482270 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.482227    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06890491-d0ee-4bc4-9e89-d7a921ed94de-model-cache\") pod \"e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb\" (UID: \"06890491-d0ee-4bc4-9e89-d7a921ed94de\") " pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:20:55.482346 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.482317    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06890491-d0ee-4bc4-9e89-d7a921ed94de-home\") pod \"e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb\" (UID: \"06890491-d0ee-4bc4-9e89-d7a921ed94de\") " pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:20:55.482411 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.482345    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06890491-d0ee-4bc4-9e89-d7a921ed94de-tmp-dir\") pod \"e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb\" (UID: \"06890491-d0ee-4bc4-9e89-d7a921ed94de\") " pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:20:55.482458 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.482441    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbj5f\" (UniqueName: \"kubernetes.io/projected/06890491-d0ee-4bc4-9e89-d7a921ed94de-kube-api-access-cbj5f\") pod \"e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb\" (UID: \"06890491-d0ee-4bc4-9e89-d7a921ed94de\") " pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:20:55.583600 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.583558    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06890491-d0ee-4bc4-9e89-d7a921ed94de-tls-certs\") pod \"e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb\" (UID: \"06890491-d0ee-4bc4-9e89-d7a921ed94de\") " pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:20:55.583600 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.583610    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06890491-d0ee-4bc4-9e89-d7a921ed94de-dshm\") pod \"e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb\" (UID: \"06890491-d0ee-4bc4-9e89-d7a921ed94de\") " pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:20:55.583861 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.583635    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06890491-d0ee-4bc4-9e89-d7a921ed94de-model-cache\") pod \"e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb\" (UID: \"06890491-d0ee-4bc4-9e89-d7a921ed94de\") " pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:20:55.583861 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.583687    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06890491-d0ee-4bc4-9e89-d7a921ed94de-home\") pod \"e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb\" (UID: \"06890491-d0ee-4bc4-9e89-d7a921ed94de\") " pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:20:55.583861 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.583708    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06890491-d0ee-4bc4-9e89-d7a921ed94de-tmp-dir\") pod \"e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb\" (UID: \"06890491-d0ee-4bc4-9e89-d7a921ed94de\") " pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:20:55.583861 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.583774    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbj5f\" (UniqueName: \"kubernetes.io/projected/06890491-d0ee-4bc4-9e89-d7a921ed94de-kube-api-access-cbj5f\") pod \"e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb\" (UID: \"06890491-d0ee-4bc4-9e89-d7a921ed94de\") " pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:20:55.584150 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.584122    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06890491-d0ee-4bc4-9e89-d7a921ed94de-home\") pod \"e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb\" (UID: \"06890491-d0ee-4bc4-9e89-d7a921ed94de\") " pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:20:55.584208 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.584144    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06890491-d0ee-4bc4-9e89-d7a921ed94de-tmp-dir\") pod \"e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb\" (UID: \"06890491-d0ee-4bc4-9e89-d7a921ed94de\") " pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:20:55.584208 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.584156    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06890491-d0ee-4bc4-9e89-d7a921ed94de-model-cache\") pod \"e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb\" (UID: \"06890491-d0ee-4bc4-9e89-d7a921ed94de\") " pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:20:55.585904 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.585879    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06890491-d0ee-4bc4-9e89-d7a921ed94de-dshm\") pod \"e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb\" (UID: \"06890491-d0ee-4bc4-9e89-d7a921ed94de\") " pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:20:55.586320 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.586300    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06890491-d0ee-4bc4-9e89-d7a921ed94de-tls-certs\") pod \"e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb\" (UID: \"06890491-d0ee-4bc4-9e89-d7a921ed94de\") " pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:20:55.595426 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.595397    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbj5f\" (UniqueName: \"kubernetes.io/projected/06890491-d0ee-4bc4-9e89-d7a921ed94de-kube-api-access-cbj5f\") pod \"e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb\" (UID: \"06890491-d0ee-4bc4-9e89-d7a921ed94de\") " pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:20:55.765659 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:55.765578    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:20:56.039161 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:56.039129    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"]
Sep 11 18:20:56.040432 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:20:56.040402    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06890491_d0ee_4bc4_9e89_d7a921ed94de.slice/crio-bb59c05c7bba030d094766ae05c3429476bcdcd196165523affb05844384f0c2 WatchSource:0}: Error finding container bb59c05c7bba030d094766ae05c3429476bcdcd196165523affb05844384f0c2: Status 404 returned error can't find the container with id bb59c05c7bba030d094766ae05c3429476bcdcd196165523affb05844384f0c2
Sep 11 18:20:56.656600 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:56.656561    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb" event={"ID":"06890491-d0ee-4bc4-9e89-d7a921ed94de","Type":"ContainerStarted","Data":"88d6c8ec95b8ed2bbd7c3b2373ab3dfe09aa0e5ae447db604fbfc0c49934c6a3"}
Sep 11 18:20:56.656600 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:56.656605    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb" event={"ID":"06890491-d0ee-4bc4-9e89-d7a921ed94de","Type":"ContainerStarted","Data":"bb59c05c7bba030d094766ae05c3429476bcdcd196165523affb05844384f0c2"}
Sep 11 18:20:56.676164 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:56.676112    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb" podStartSLOduration=1.676096833 podStartE2EDuration="1.676096833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:20:56.67342518 +0000 UTC m=+2162.118525751" watchObservedRunningTime="2026-09-11 18:20:56.676096833 +0000 UTC m=+2162.121197405"
Sep 11 18:20:57.825112 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:20:57.825073    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:21:01.369953 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:01.369911    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"]
Sep 11 18:21:01.373317 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:01.373298    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:01.378967 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:01.378946    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-b14014df-kserve-self-signed-certs\""
Sep 11 18:21:01.388620 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:01.388592    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"]
Sep 11 18:21:01.438441 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:01.438404    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f154424-e768-49ca-a6bc-2ec84bf78051-tls-certs\") pod \"e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z\" (UID: \"5f154424-e768-49ca-a6bc-2ec84bf78051\") " pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:01.438631 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:01.438452    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f154424-e768-49ca-a6bc-2ec84bf78051-model-cache\") pod \"e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z\" (UID: \"5f154424-e768-49ca-a6bc-2ec84bf78051\") " pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:01.438631 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:01.438484    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f154424-e768-49ca-a6bc-2ec84bf78051-home\") pod \"e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z\" (UID: \"5f154424-e768-49ca-a6bc-2ec84bf78051\") " pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:01.438631 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:01.438519    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f154424-e768-49ca-a6bc-2ec84bf78051-tmp-dir\") pod \"e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z\" (UID: \"5f154424-e768-49ca-a6bc-2ec84bf78051\") " pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:01.438631 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:01.438541    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f154424-e768-49ca-a6bc-2ec84bf78051-dshm\") pod \"e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z\" (UID: \"5f154424-e768-49ca-a6bc-2ec84bf78051\") " pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:01.438631 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:01.438601    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t6zk\" (UniqueName: \"kubernetes.io/projected/5f154424-e768-49ca-a6bc-2ec84bf78051-kube-api-access-7t6zk\") pod \"e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z\" (UID: \"5f154424-e768-49ca-a6bc-2ec84bf78051\") " pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:01.539603 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:01.539564    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f154424-e768-49ca-a6bc-2ec84bf78051-tls-certs\") pod \"e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z\" (UID: \"5f154424-e768-49ca-a6bc-2ec84bf78051\") " pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:01.539816 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:01.539612    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f154424-e768-49ca-a6bc-2ec84bf78051-model-cache\") pod \"e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z\" (UID: \"5f154424-e768-49ca-a6bc-2ec84bf78051\") " pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:01.539816 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:01.539646    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f154424-e768-49ca-a6bc-2ec84bf78051-home\") pod \"e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z\" (UID: \"5f154424-e768-49ca-a6bc-2ec84bf78051\") " pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:01.539816 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:01.539669    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f154424-e768-49ca-a6bc-2ec84bf78051-tmp-dir\") pod \"e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z\" (UID: \"5f154424-e768-49ca-a6bc-2ec84bf78051\") " pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:01.539816 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:01.539688    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f154424-e768-49ca-a6bc-2ec84bf78051-dshm\") pod \"e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z\" (UID: \"5f154424-e768-49ca-a6bc-2ec84bf78051\") " pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:01.539816 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:01.539754    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t6zk\" (UniqueName: \"kubernetes.io/projected/5f154424-e768-49ca-a6bc-2ec84bf78051-kube-api-access-7t6zk\") pod \"e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z\" (UID: \"5f154424-e768-49ca-a6bc-2ec84bf78051\") " pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:01.540208 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:01.540156    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f154424-e768-49ca-a6bc-2ec84bf78051-home\") pod \"e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z\" (UID: \"5f154424-e768-49ca-a6bc-2ec84bf78051\") " pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:01.540363 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:01.540185    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f154424-e768-49ca-a6bc-2ec84bf78051-tmp-dir\") pod \"e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z\" (UID: \"5f154424-e768-49ca-a6bc-2ec84bf78051\") " pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:01.540363 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:01.540240    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f154424-e768-49ca-a6bc-2ec84bf78051-model-cache\") pod \"e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z\" (UID: \"5f154424-e768-49ca-a6bc-2ec84bf78051\") " pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:01.541956 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:01.541935    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f154424-e768-49ca-a6bc-2ec84bf78051-dshm\") pod \"e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z\" (UID: \"5f154424-e768-49ca-a6bc-2ec84bf78051\") " pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:01.542234 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:01.542215    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f154424-e768-49ca-a6bc-2ec84bf78051-tls-certs\") pod \"e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z\" (UID: \"5f154424-e768-49ca-a6bc-2ec84bf78051\") " pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:01.550751 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:01.550725    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t6zk\" (UniqueName: \"kubernetes.io/projected/5f154424-e768-49ca-a6bc-2ec84bf78051-kube-api-access-7t6zk\") pod \"e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z\" (UID: \"5f154424-e768-49ca-a6bc-2ec84bf78051\") " pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:01.683134 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:01.683098    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:01.964285 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:01.964257    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"]
Sep 11 18:21:01.966363 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:21:01.966321    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f154424_e768_49ca_a6bc_2ec84bf78051.slice/crio-60c97d677059c8334e613e0fafef097d1d28417e6cd30f4659cf3ddc7ba829ff WatchSource:0}: Error finding container 60c97d677059c8334e613e0fafef097d1d28417e6cd30f4659cf3ddc7ba829ff: Status 404 returned error can't find the container with id 60c97d677059c8334e613e0fafef097d1d28417e6cd30f4659cf3ddc7ba829ff
Sep 11 18:21:02.691034 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:02.690995    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z" event={"ID":"5f154424-e768-49ca-a6bc-2ec84bf78051","Type":"ContainerStarted","Data":"2bf8d873f3b629f03b6b5f2ba91fc004c4f8c8c5805d7e2a59bc6fa58666f3c3"}
Sep 11 18:21:02.691511 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:02.691043    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z" event={"ID":"5f154424-e768-49ca-a6bc-2ec84bf78051","Type":"ContainerStarted","Data":"60c97d677059c8334e613e0fafef097d1d28417e6cd30f4659cf3ddc7ba829ff"}
Sep 11 18:21:02.711177 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:02.711125    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z" podStartSLOduration=1.711109255 podStartE2EDuration="1.711109255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:21:02.709588971 +0000 UTC m=+2168.154689543" watchObservedRunningTime="2026-09-11 18:21:02.711109255 +0000 UTC m=+2168.156209825"
Sep 11 18:21:05.158747 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:05.158712    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:21:05.765933 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:05.765896    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:21:05.766103 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:05.765957    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:21:05.778674 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:05.778638    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:21:06.723804 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:06.723766    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:21:11.683687 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:11.683634    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:11.684097 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:11.683701    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:11.696216 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:11.696191    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:11.748870 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:11.748838    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:16.835080 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:16.835047    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"]
Sep 11 18:21:18.827312 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:18.827277    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"]
Sep 11 18:21:18.867020 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:18.866980    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:21:26.639095 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:26.639056    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:21:31.854112 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:31.854066    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb" podUID="06890491-d0ee-4bc4-9e89-d7a921ed94de" containerName="main" containerID="cri-o://88d6c8ec95b8ed2bbd7c3b2373ab3dfe09aa0e5ae447db604fbfc0c49934c6a3" gracePeriod=45
Sep 11 18:21:32.109946 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.109883    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:21:32.127297 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.127269    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbj5f\" (UniqueName: \"kubernetes.io/projected/06890491-d0ee-4bc4-9e89-d7a921ed94de-kube-api-access-cbj5f\") pod \"06890491-d0ee-4bc4-9e89-d7a921ed94de\" (UID: \"06890491-d0ee-4bc4-9e89-d7a921ed94de\") "
Sep 11 18:21:32.127425 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.127319    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06890491-d0ee-4bc4-9e89-d7a921ed94de-tmp-dir\") pod \"06890491-d0ee-4bc4-9e89-d7a921ed94de\" (UID: \"06890491-d0ee-4bc4-9e89-d7a921ed94de\") "
Sep 11 18:21:32.127425 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.127341    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06890491-d0ee-4bc4-9e89-d7a921ed94de-home\") pod \"06890491-d0ee-4bc4-9e89-d7a921ed94de\" (UID: \"06890491-d0ee-4bc4-9e89-d7a921ed94de\") "
Sep 11 18:21:32.127505 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.127458    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06890491-d0ee-4bc4-9e89-d7a921ed94de-tls-certs\") pod \"06890491-d0ee-4bc4-9e89-d7a921ed94de\" (UID: \"06890491-d0ee-4bc4-9e89-d7a921ed94de\") "
Sep 11 18:21:32.127505 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.127478    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06890491-d0ee-4bc4-9e89-d7a921ed94de-model-cache\") pod \"06890491-d0ee-4bc4-9e89-d7a921ed94de\" (UID: \"06890491-d0ee-4bc4-9e89-d7a921ed94de\") "
Sep 11 18:21:32.127505 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.127499    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06890491-d0ee-4bc4-9e89-d7a921ed94de-dshm\") pod \"06890491-d0ee-4bc4-9e89-d7a921ed94de\" (UID: \"06890491-d0ee-4bc4-9e89-d7a921ed94de\") "
Sep 11 18:21:32.127733 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.127625    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06890491-d0ee-4bc4-9e89-d7a921ed94de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06890491-d0ee-4bc4-9e89-d7a921ed94de" (UID: "06890491-d0ee-4bc4-9e89-d7a921ed94de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:32.127733 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.127702    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06890491-d0ee-4bc4-9e89-d7a921ed94de-home" (OuterVolumeSpecName: "home") pod "06890491-d0ee-4bc4-9e89-d7a921ed94de" (UID: "06890491-d0ee-4bc4-9e89-d7a921ed94de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:32.127845 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.127805    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06890491-d0ee-4bc4-9e89-d7a921ed94de-model-cache" (OuterVolumeSpecName: "model-cache") pod "06890491-d0ee-4bc4-9e89-d7a921ed94de" (UID: "06890491-d0ee-4bc4-9e89-d7a921ed94de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:32.127903 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.127851    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06890491-d0ee-4bc4-9e89-d7a921ed94de-tmp-dir\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:21:32.127903 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.127871    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06890491-d0ee-4bc4-9e89-d7a921ed94de-home\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:21:32.129847 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.129818    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06890491-d0ee-4bc4-9e89-d7a921ed94de-kube-api-access-cbj5f" (OuterVolumeSpecName: "kube-api-access-cbj5f") pod "06890491-d0ee-4bc4-9e89-d7a921ed94de" (UID: "06890491-d0ee-4bc4-9e89-d7a921ed94de"). InnerVolumeSpecName "kube-api-access-cbj5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:21:32.130323 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.130294    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06890491-d0ee-4bc4-9e89-d7a921ed94de-dshm" (OuterVolumeSpecName: "dshm") pod "06890491-d0ee-4bc4-9e89-d7a921ed94de" (UID: "06890491-d0ee-4bc4-9e89-d7a921ed94de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:32.130543 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.130517    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06890491-d0ee-4bc4-9e89-d7a921ed94de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06890491-d0ee-4bc4-9e89-d7a921ed94de" (UID: "06890491-d0ee-4bc4-9e89-d7a921ed94de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:21:32.228848 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.228808    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbj5f\" (UniqueName: \"kubernetes.io/projected/06890491-d0ee-4bc4-9e89-d7a921ed94de-kube-api-access-cbj5f\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:21:32.228848 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.228843    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06890491-d0ee-4bc4-9e89-d7a921ed94de-tls-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:21:32.228848 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.228853    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06890491-d0ee-4bc4-9e89-d7a921ed94de-model-cache\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:21:32.229086 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.228861    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06890491-d0ee-4bc4-9e89-d7a921ed94de-dshm\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:21:32.437743 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.437706    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:21:32.853615 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.853520    2559 generic.go:356] "Generic (PLEG): container finished" podID="06890491-d0ee-4bc4-9e89-d7a921ed94de" containerID="88d6c8ec95b8ed2bbd7c3b2373ab3dfe09aa0e5ae447db604fbfc0c49934c6a3" exitCode=0
Sep 11 18:21:32.853615 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.853567    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"
Sep 11 18:21:32.853814 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.853607    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb" event={"ID":"06890491-d0ee-4bc4-9e89-d7a921ed94de","Type":"ContainerDied","Data":"88d6c8ec95b8ed2bbd7c3b2373ab3dfe09aa0e5ae447db604fbfc0c49934c6a3"}
Sep 11 18:21:32.853814 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.853648    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb" event={"ID":"06890491-d0ee-4bc4-9e89-d7a921ed94de","Type":"ContainerDied","Data":"bb59c05c7bba030d094766ae05c3429476bcdcd196165523affb05844384f0c2"}
Sep 11 18:21:32.853814 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.853665    2559 scope.go:117] "RemoveContainer" containerID="88d6c8ec95b8ed2bbd7c3b2373ab3dfe09aa0e5ae447db604fbfc0c49934c6a3"
Sep 11 18:21:32.863079 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.863062    2559 scope.go:117] "RemoveContainer" containerID="88d6c8ec95b8ed2bbd7c3b2373ab3dfe09aa0e5ae447db604fbfc0c49934c6a3"
Sep 11 18:21:32.863334 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:21:32.863324    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d6c8ec95b8ed2bbd7c3b2373ab3dfe09aa0e5ae447db604fbfc0c49934c6a3\": container with ID starting with 88d6c8ec95b8ed2bbd7c3b2373ab3dfe09aa0e5ae447db604fbfc0c49934c6a3 not found: ID does not exist" containerID="88d6c8ec95b8ed2bbd7c3b2373ab3dfe09aa0e5ae447db604fbfc0c49934c6a3"
Sep 11 18:21:32.863392 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.863359    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d6c8ec95b8ed2bbd7c3b2373ab3dfe09aa0e5ae447db604fbfc0c49934c6a3"} err="failed to get container status \"88d6c8ec95b8ed2bbd7c3b2373ab3dfe09aa0e5ae447db604fbfc0c49934c6a3\": rpc error: code = NotFound desc = could not find container \"88d6c8ec95b8ed2bbd7c3b2373ab3dfe09aa0e5ae447db604fbfc0c49934c6a3\": container with ID starting with 88d6c8ec95b8ed2bbd7c3b2373ab3dfe09aa0e5ae447db604fbfc0c49934c6a3 not found: ID does not exist"
Sep 11 18:21:32.878838 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.878810    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"]
Sep 11 18:21:32.881008 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:32.880989    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-b14014df-kserve-6cbf48c67-mm8pb"]
Sep 11 18:21:33.052803 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:33.052770    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06890491-d0ee-4bc4-9e89-d7a921ed94de" path="/var/lib/kubelet/pods/06890491-d0ee-4bc4-9e89-d7a921ed94de/volumes"
Sep 11 18:21:33.846402 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:33.846293    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z" podUID="5f154424-e768-49ca-a6bc-2ec84bf78051" containerName="main" containerID="cri-o://2bf8d873f3b629f03b6b5f2ba91fc004c4f8c8c5805d7e2a59bc6fa58666f3c3" gracePeriod=45
Sep 11 18:21:34.102135 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.102072    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:34.144822 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.144789    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f154424-e768-49ca-a6bc-2ec84bf78051-home\") pod \"5f154424-e768-49ca-a6bc-2ec84bf78051\" (UID: \"5f154424-e768-49ca-a6bc-2ec84bf78051\") "
Sep 11 18:21:34.144997 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.144850    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f154424-e768-49ca-a6bc-2ec84bf78051-tls-certs\") pod \"5f154424-e768-49ca-a6bc-2ec84bf78051\" (UID: \"5f154424-e768-49ca-a6bc-2ec84bf78051\") "
Sep 11 18:21:34.144997 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.144877    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f154424-e768-49ca-a6bc-2ec84bf78051-tmp-dir\") pod \"5f154424-e768-49ca-a6bc-2ec84bf78051\" (UID: \"5f154424-e768-49ca-a6bc-2ec84bf78051\") "
Sep 11 18:21:34.144997 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.144901    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f154424-e768-49ca-a6bc-2ec84bf78051-model-cache\") pod \"5f154424-e768-49ca-a6bc-2ec84bf78051\" (UID: \"5f154424-e768-49ca-a6bc-2ec84bf78051\") "
Sep 11 18:21:34.144997 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.144927    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7t6zk\" (UniqueName: \"kubernetes.io/projected/5f154424-e768-49ca-a6bc-2ec84bf78051-kube-api-access-7t6zk\") pod \"5f154424-e768-49ca-a6bc-2ec84bf78051\" (UID: \"5f154424-e768-49ca-a6bc-2ec84bf78051\") "
Sep 11 18:21:34.144997 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.144968    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f154424-e768-49ca-a6bc-2ec84bf78051-dshm\") pod \"5f154424-e768-49ca-a6bc-2ec84bf78051\" (UID: \"5f154424-e768-49ca-a6bc-2ec84bf78051\") "
Sep 11 18:21:34.145273 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.145056    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f154424-e768-49ca-a6bc-2ec84bf78051-home" (OuterVolumeSpecName: "home") pod "5f154424-e768-49ca-a6bc-2ec84bf78051" (UID: "5f154424-e768-49ca-a6bc-2ec84bf78051"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:34.145273 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.145160    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f154424-e768-49ca-a6bc-2ec84bf78051-model-cache" (OuterVolumeSpecName: "model-cache") pod "5f154424-e768-49ca-a6bc-2ec84bf78051" (UID: "5f154424-e768-49ca-a6bc-2ec84bf78051"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:34.145273 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.145193    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f154424-e768-49ca-a6bc-2ec84bf78051-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5f154424-e768-49ca-a6bc-2ec84bf78051" (UID: "5f154424-e768-49ca-a6bc-2ec84bf78051"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:34.145438 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.145325    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f154424-e768-49ca-a6bc-2ec84bf78051-tmp-dir\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:21:34.145438 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.145344    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f154424-e768-49ca-a6bc-2ec84bf78051-model-cache\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:21:34.145438 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.145378    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f154424-e768-49ca-a6bc-2ec84bf78051-home\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:21:34.147075 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.147048    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f154424-e768-49ca-a6bc-2ec84bf78051-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f154424-e768-49ca-a6bc-2ec84bf78051" (UID: "5f154424-e768-49ca-a6bc-2ec84bf78051"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:21:34.147211 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.147170    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f154424-e768-49ca-a6bc-2ec84bf78051-dshm" (OuterVolumeSpecName: "dshm") pod "5f154424-e768-49ca-a6bc-2ec84bf78051" (UID: "5f154424-e768-49ca-a6bc-2ec84bf78051"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:34.147544 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.147523    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f154424-e768-49ca-a6bc-2ec84bf78051-kube-api-access-7t6zk" (OuterVolumeSpecName: "kube-api-access-7t6zk") pod "5f154424-e768-49ca-a6bc-2ec84bf78051" (UID: "5f154424-e768-49ca-a6bc-2ec84bf78051"). InnerVolumeSpecName "kube-api-access-7t6zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:21:34.246518 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.246469    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f154424-e768-49ca-a6bc-2ec84bf78051-tls-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:21:34.246518 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.246506    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t6zk\" (UniqueName: \"kubernetes.io/projected/5f154424-e768-49ca-a6bc-2ec84bf78051-kube-api-access-7t6zk\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:21:34.246518 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.246517    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f154424-e768-49ca-a6bc-2ec84bf78051-dshm\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:21:34.865606 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.865566    2559 generic.go:356] "Generic (PLEG): container finished" podID="5f154424-e768-49ca-a6bc-2ec84bf78051" containerID="2bf8d873f3b629f03b6b5f2ba91fc004c4f8c8c5805d7e2a59bc6fa58666f3c3" exitCode=0
Sep 11 18:21:34.865796 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.865623    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"
Sep 11 18:21:34.865796 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.865666    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z" event={"ID":"5f154424-e768-49ca-a6bc-2ec84bf78051","Type":"ContainerDied","Data":"2bf8d873f3b629f03b6b5f2ba91fc004c4f8c8c5805d7e2a59bc6fa58666f3c3"}
Sep 11 18:21:34.865796 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.865718    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z" event={"ID":"5f154424-e768-49ca-a6bc-2ec84bf78051","Type":"ContainerDied","Data":"60c97d677059c8334e613e0fafef097d1d28417e6cd30f4659cf3ddc7ba829ff"}
Sep 11 18:21:34.865796 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.865742    2559 scope.go:117] "RemoveContainer" containerID="2bf8d873f3b629f03b6b5f2ba91fc004c4f8c8c5805d7e2a59bc6fa58666f3c3"
Sep 11 18:21:34.881808 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.881786    2559 scope.go:117] "RemoveContainer" containerID="2bf8d873f3b629f03b6b5f2ba91fc004c4f8c8c5805d7e2a59bc6fa58666f3c3"
Sep 11 18:21:34.882106 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:21:34.882087    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf8d873f3b629f03b6b5f2ba91fc004c4f8c8c5805d7e2a59bc6fa58666f3c3\": container with ID starting with 2bf8d873f3b629f03b6b5f2ba91fc004c4f8c8c5805d7e2a59bc6fa58666f3c3 not found: ID does not exist" containerID="2bf8d873f3b629f03b6b5f2ba91fc004c4f8c8c5805d7e2a59bc6fa58666f3c3"
Sep 11 18:21:34.882153 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.882114    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf8d873f3b629f03b6b5f2ba91fc004c4f8c8c5805d7e2a59bc6fa58666f3c3"} err="failed to get container status \"2bf8d873f3b629f03b6b5f2ba91fc004c4f8c8c5805d7e2a59bc6fa58666f3c3\": rpc error: code = NotFound desc = could not find container \"2bf8d873f3b629f03b6b5f2ba91fc004c4f8c8c5805d7e2a59bc6fa58666f3c3\": container with ID starting with 2bf8d873f3b629f03b6b5f2ba91fc004c4f8c8c5805d7e2a59bc6fa58666f3c3 not found: ID does not exist"
Sep 11 18:21:34.895735 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.895700    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"]
Sep 11 18:21:34.900008 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:34.899977    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-b14014df-kserve-7fb5d6b9d7-q5h9z"]
Sep 11 18:21:35.052421 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:35.052386    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f154424-e768-49ca-a6bc-2ec84bf78051" path="/var/lib/kubelet/pods/5f154424-e768-49ca-a6bc-2ec84bf78051/volumes"
Sep 11 18:21:40.031550 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.031508    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:21:40.459566 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.459520    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk"]
Sep 11 18:21:40.459991 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.459971    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f154424-e768-49ca-a6bc-2ec84bf78051" containerName="main"
Sep 11 18:21:40.460110 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.459993    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f154424-e768-49ca-a6bc-2ec84bf78051" containerName="main"
Sep 11 18:21:40.460110 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.460017    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06890491-d0ee-4bc4-9e89-d7a921ed94de" containerName="main"
Sep 11 18:21:40.460110 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.460025    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="06890491-d0ee-4bc4-9e89-d7a921ed94de" containerName="main"
Sep 11 18:21:40.460269 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.460114    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="06890491-d0ee-4bc4-9e89-d7a921ed94de" containerName="main"
Sep 11 18:21:40.460269 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.460128    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f154424-e768-49ca-a6bc-2ec84bf78051" containerName="main"
Sep 11 18:21:40.464485 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.464457    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk"
Sep 11 18:21:40.467730 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.467706    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 18:21:40.470056 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.470032    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 18:21:40.470394 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.470374    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cdcpv\""
Sep 11 18:21:40.471739 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.471717    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:21:40.474993 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.474975    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 18:21:40.487342 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.487319    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk"]
Sep 11 18:21:40.500504 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.500478    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf2ws\" (UniqueName: \"kubernetes.io/projected/4d92c850-d415-4bc7-9fb2-34065e6411de-kube-api-access-lf2ws\") pod \"maas-api-55bbb86db8-k74jk\" (UID: \"4d92c850-d415-4bc7-9fb2-34065e6411de\") " pod="odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk"
Sep 11 18:21:40.500643 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.500536    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d92c850-d415-4bc7-9fb2-34065e6411de-openshift-service-ca\") pod \"maas-api-55bbb86db8-k74jk\" (UID: \"4d92c850-d415-4bc7-9fb2-34065e6411de\") " pod="odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk"
Sep 11 18:21:40.500688 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.500648    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d92c850-d415-4bc7-9fb2-34065e6411de-maas-api-tls\") pod \"maas-api-55bbb86db8-k74jk\" (UID: \"4d92c850-d415-4bc7-9fb2-34065e6411de\") " pod="odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk"
Sep 11 18:21:40.500725 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.500699    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d92c850-d415-4bc7-9fb2-34065e6411de-metrics-certs\") pod \"maas-api-55bbb86db8-k74jk\" (UID: \"4d92c850-d415-4bc7-9fb2-34065e6411de\") " pod="odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk"
Sep 11 18:21:40.601411 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.601345    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d92c850-d415-4bc7-9fb2-34065e6411de-openshift-service-ca\") pod \"maas-api-55bbb86db8-k74jk\" (UID: \"4d92c850-d415-4bc7-9fb2-34065e6411de\") " pod="odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk"
Sep 11 18:21:40.601600 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.601518    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d92c850-d415-4bc7-9fb2-34065e6411de-maas-api-tls\") pod \"maas-api-55bbb86db8-k74jk\" (UID: \"4d92c850-d415-4bc7-9fb2-34065e6411de\") " pod="odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk"
Sep 11 18:21:40.601600 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.601549    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d92c850-d415-4bc7-9fb2-34065e6411de-metrics-certs\") pod \"maas-api-55bbb86db8-k74jk\" (UID: \"4d92c850-d415-4bc7-9fb2-34065e6411de\") " pod="odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk"
Sep 11 18:21:40.601680 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.601620    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf2ws\" (UniqueName: \"kubernetes.io/projected/4d92c850-d415-4bc7-9fb2-34065e6411de-kube-api-access-lf2ws\") pod \"maas-api-55bbb86db8-k74jk\" (UID: \"4d92c850-d415-4bc7-9fb2-34065e6411de\") " pod="odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk"
Sep 11 18:21:40.602116 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.602099    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d92c850-d415-4bc7-9fb2-34065e6411de-openshift-service-ca\") pod \"maas-api-55bbb86db8-k74jk\" (UID: \"4d92c850-d415-4bc7-9fb2-34065e6411de\") " pod="odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk"
Sep 11 18:21:40.604043 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.604016    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d92c850-d415-4bc7-9fb2-34065e6411de-metrics-certs\") pod \"maas-api-55bbb86db8-k74jk\" (UID: \"4d92c850-d415-4bc7-9fb2-34065e6411de\") " pod="odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk"
Sep 11 18:21:40.604163 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.604078    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d92c850-d415-4bc7-9fb2-34065e6411de-maas-api-tls\") pod \"maas-api-55bbb86db8-k74jk\" (UID: \"4d92c850-d415-4bc7-9fb2-34065e6411de\") " pod="odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk"
Sep 11 18:21:40.622283 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.622259    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf2ws\" (UniqueName: \"kubernetes.io/projected/4d92c850-d415-4bc7-9fb2-34065e6411de-kube-api-access-lf2ws\") pod \"maas-api-55bbb86db8-k74jk\" (UID: \"4d92c850-d415-4bc7-9fb2-34065e6411de\") " pod="odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk"
Sep 11 18:21:40.775241 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:40.775143    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk"
Sep 11 18:21:41.057167 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:41.057138    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk"]
Sep 11 18:21:41.059202 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:21:41.059179    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d92c850_d415_4bc7_9fb2_34065e6411de.slice/crio-4f46c91966d5bc57da36b3516d9e876fb6b0d749308cdd37cb638ddb73d64fcc WatchSource:0}: Error finding container 4f46c91966d5bc57da36b3516d9e876fb6b0d749308cdd37cb638ddb73d64fcc: Status 404 returned error can't find the container with id 4f46c91966d5bc57da36b3516d9e876fb6b0d749308cdd37cb638ddb73d64fcc
Sep 11 18:21:41.060493 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:41.060474    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:21:41.903873 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:41.903833    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk" event={"ID":"4d92c850-d415-4bc7-9fb2-34065e6411de","Type":"ContainerStarted","Data":"e18621a8dead747532c96709ec207a73c06c464303e21760715c79010603d1fa"}
Sep 11 18:21:41.904074 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:41.903912    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk" event={"ID":"4d92c850-d415-4bc7-9fb2-34065e6411de","Type":"ContainerStarted","Data":"4f46c91966d5bc57da36b3516d9e876fb6b0d749308cdd37cb638ddb73d64fcc"}
Sep 11 18:21:41.904074 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:41.904055    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk"
Sep 11 18:21:41.933288 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:41.933238    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk" podStartSLOduration=1.507142684 podStartE2EDuration="1.933220438s" totalImagesPullingTime="426.077754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:21:41.930872326 +0000 UTC m=+2207.375972896" watchObservedRunningTime="2026-09-11 18:21:41.933220438 +0000 UTC m=+2207.378321008"
Sep 11 18:21:47.914983 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:47.914953    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk"
Sep 11 18:21:48.246607 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:48.246524    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:21:48.528425 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:48.528332    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c66b56868-mwhzm"]
Sep 11 18:21:48.533342 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:48.533314    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c66b56868-mwhzm"
Sep 11 18:21:48.544886 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:48.544855    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c66b56868-mwhzm"]
Sep 11 18:21:48.574539 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:48.574505    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/224a4296-05d2-4221-a735-aa666ac1a846-metrics-certs\") pod \"maas-api-5c66b56868-mwhzm\" (UID: \"224a4296-05d2-4221-a735-aa666ac1a846\") " pod="odh-ai-gateway-infra/maas-api-5c66b56868-mwhzm"
Sep 11 18:21:48.574694 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:48.574573    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/224a4296-05d2-4221-a735-aa666ac1a846-openshift-service-ca\") pod \"maas-api-5c66b56868-mwhzm\" (UID: \"224a4296-05d2-4221-a735-aa666ac1a846\") " pod="odh-ai-gateway-infra/maas-api-5c66b56868-mwhzm"
Sep 11 18:21:48.574694 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:48.574604    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch65k\" (UniqueName: \"kubernetes.io/projected/224a4296-05d2-4221-a735-aa666ac1a846-kube-api-access-ch65k\") pod \"maas-api-5c66b56868-mwhzm\" (UID: \"224a4296-05d2-4221-a735-aa666ac1a846\") " pod="odh-ai-gateway-infra/maas-api-5c66b56868-mwhzm"
Sep 11 18:21:48.574694 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:48.574636    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/224a4296-05d2-4221-a735-aa666ac1a846-maas-api-tls\") pod \"maas-api-5c66b56868-mwhzm\" (UID: \"224a4296-05d2-4221-a735-aa666ac1a846\") " pod="odh-ai-gateway-infra/maas-api-5c66b56868-mwhzm"
Sep 11 18:21:48.675809 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:48.675768    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/224a4296-05d2-4221-a735-aa666ac1a846-openshift-service-ca\") pod \"maas-api-5c66b56868-mwhzm\" (UID: \"224a4296-05d2-4221-a735-aa666ac1a846\") " pod="odh-ai-gateway-infra/maas-api-5c66b56868-mwhzm"
Sep 11 18:21:48.675997 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:48.675819    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch65k\" (UniqueName: \"kubernetes.io/projected/224a4296-05d2-4221-a735-aa666ac1a846-kube-api-access-ch65k\") pod \"maas-api-5c66b56868-mwhzm\" (UID: \"224a4296-05d2-4221-a735-aa666ac1a846\") " pod="odh-ai-gateway-infra/maas-api-5c66b56868-mwhzm"
Sep 11 18:21:48.675997 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:48.675866    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/224a4296-05d2-4221-a735-aa666ac1a846-maas-api-tls\") pod \"maas-api-5c66b56868-mwhzm\" (UID: \"224a4296-05d2-4221-a735-aa666ac1a846\") " pod="odh-ai-gateway-infra/maas-api-5c66b56868-mwhzm"
Sep 11 18:21:48.675997 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:48.675928    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/224a4296-05d2-4221-a735-aa666ac1a846-metrics-certs\") pod \"maas-api-5c66b56868-mwhzm\" (UID: \"224a4296-05d2-4221-a735-aa666ac1a846\") " pod="odh-ai-gateway-infra/maas-api-5c66b56868-mwhzm"
Sep 11 18:21:48.676628 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:48.676604    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/224a4296-05d2-4221-a735-aa666ac1a846-openshift-service-ca\") pod \"maas-api-5c66b56868-mwhzm\" (UID: \"224a4296-05d2-4221-a735-aa666ac1a846\") " pod="odh-ai-gateway-infra/maas-api-5c66b56868-mwhzm"
Sep 11 18:21:48.678378 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:48.678338    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/224a4296-05d2-4221-a735-aa666ac1a846-maas-api-tls\") pod \"maas-api-5c66b56868-mwhzm\" (UID: \"224a4296-05d2-4221-a735-aa666ac1a846\") " pod="odh-ai-gateway-infra/maas-api-5c66b56868-mwhzm"
Sep 11 18:21:48.678470 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:48.678411    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/224a4296-05d2-4221-a735-aa666ac1a846-metrics-certs\") pod \"maas-api-5c66b56868-mwhzm\" (UID: \"224a4296-05d2-4221-a735-aa666ac1a846\") " pod="odh-ai-gateway-infra/maas-api-5c66b56868-mwhzm"
Sep 11 18:21:48.687058 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:48.687026    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch65k\" (UniqueName: \"kubernetes.io/projected/224a4296-05d2-4221-a735-aa666ac1a846-kube-api-access-ch65k\") pod \"maas-api-5c66b56868-mwhzm\" (UID: \"224a4296-05d2-4221-a735-aa666ac1a846\") " pod="odh-ai-gateway-infra/maas-api-5c66b56868-mwhzm"
Sep 11 18:21:48.846584 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:48.846480    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c66b56868-mwhzm"
Sep 11 18:21:49.125147 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:49.125115    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c66b56868-mwhzm"]
Sep 11 18:21:49.125803 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:21:49.125775    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224a4296_05d2_4221_a735_aa666ac1a846.slice/crio-3d122af759a78689f75d722797b16bd416ad50dd5b224f7c587c474532054f9b WatchSource:0}: Error finding container 3d122af759a78689f75d722797b16bd416ad50dd5b224f7c587c474532054f9b: Status 404 returned error can't find the container with id 3d122af759a78689f75d722797b16bd416ad50dd5b224f7c587c474532054f9b
Sep 11 18:21:49.948399 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:49.948342    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c66b56868-mwhzm" event={"ID":"224a4296-05d2-4221-a735-aa666ac1a846","Type":"ContainerStarted","Data":"a0fc34c65c2c0e6f0ad88e525d4f08d7922a621290f33ca572e577c9fc89c6c1"}
Sep 11 18:21:49.948588 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:49.948425    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c66b56868-mwhzm" event={"ID":"224a4296-05d2-4221-a735-aa666ac1a846","Type":"ContainerStarted","Data":"3d122af759a78689f75d722797b16bd416ad50dd5b224f7c587c474532054f9b"}
Sep 11 18:21:49.948588 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:49.948500    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c66b56868-mwhzm"
Sep 11 18:21:49.971225 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:49.971177    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c66b56868-mwhzm" podStartSLOduration=1.5407362820000001 podStartE2EDuration="1.97116122s" totalImagesPullingTime="430.424938ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:21:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:21:49.970371994 +0000 UTC m=+2215.415472559" watchObservedRunningTime="2026-09-11 18:21:49.97116122 +0000 UTC m=+2215.416261791"
Sep 11 18:21:55.130954 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:55.130916    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:21:55.960369 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:55.960316    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c66b56868-mwhzm"
Sep 11 18:21:56.031168 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:56.031133    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk"]
Sep 11 18:21:56.031399 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:56.031371    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk" podUID="4d92c850-d415-4bc7-9fb2-34065e6411de" containerName="maas-api" containerID="cri-o://e18621a8dead747532c96709ec207a73c06c464303e21760715c79010603d1fa" gracePeriod=30
Sep 11 18:21:57.277865 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:57.277843    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk"
Sep 11 18:21:57.362395 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:57.362298    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d92c850-d415-4bc7-9fb2-34065e6411de-metrics-certs\") pod \"4d92c850-d415-4bc7-9fb2-34065e6411de\" (UID: \"4d92c850-d415-4bc7-9fb2-34065e6411de\") "
Sep 11 18:21:57.362395 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:57.362365    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lf2ws\" (UniqueName: \"kubernetes.io/projected/4d92c850-d415-4bc7-9fb2-34065e6411de-kube-api-access-lf2ws\") pod \"4d92c850-d415-4bc7-9fb2-34065e6411de\" (UID: \"4d92c850-d415-4bc7-9fb2-34065e6411de\") "
Sep 11 18:21:57.362395 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:57.362397    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d92c850-d415-4bc7-9fb2-34065e6411de-maas-api-tls\") pod \"4d92c850-d415-4bc7-9fb2-34065e6411de\" (UID: \"4d92c850-d415-4bc7-9fb2-34065e6411de\") "
Sep 11 18:21:57.362614 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:57.362442    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d92c850-d415-4bc7-9fb2-34065e6411de-openshift-service-ca\") pod \"4d92c850-d415-4bc7-9fb2-34065e6411de\" (UID: \"4d92c850-d415-4bc7-9fb2-34065e6411de\") "
Sep 11 18:21:57.362876 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:57.362846    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d92c850-d415-4bc7-9fb2-34065e6411de-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4d92c850-d415-4bc7-9fb2-34065e6411de" (UID: "4d92c850-d415-4bc7-9fb2-34065e6411de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:21:57.364587 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:57.364560    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d92c850-d415-4bc7-9fb2-34065e6411de-kube-api-access-lf2ws" (OuterVolumeSpecName: "kube-api-access-lf2ws") pod "4d92c850-d415-4bc7-9fb2-34065e6411de" (UID: "4d92c850-d415-4bc7-9fb2-34065e6411de"). InnerVolumeSpecName "kube-api-access-lf2ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:21:57.364692 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:57.364614    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d92c850-d415-4bc7-9fb2-34065e6411de-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4d92c850-d415-4bc7-9fb2-34065e6411de" (UID: "4d92c850-d415-4bc7-9fb2-34065e6411de"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:21:57.364692 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:57.364631    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d92c850-d415-4bc7-9fb2-34065e6411de-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4d92c850-d415-4bc7-9fb2-34065e6411de" (UID: "4d92c850-d415-4bc7-9fb2-34065e6411de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:21:57.463725 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:57.463693    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d92c850-d415-4bc7-9fb2-34065e6411de-metrics-certs\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:21:57.463725 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:57.463722    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf2ws\" (UniqueName: \"kubernetes.io/projected/4d92c850-d415-4bc7-9fb2-34065e6411de-kube-api-access-lf2ws\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:21:57.463725 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:57.463733    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d92c850-d415-4bc7-9fb2-34065e6411de-maas-api-tls\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:21:57.463946 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:57.463744    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d92c850-d415-4bc7-9fb2-34065e6411de-openshift-service-ca\") on node \"ip-10-0-138-96.ec2.internal\" DevicePath \"\""
Sep 11 18:21:57.998219 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:57.998177    2559 generic.go:356] "Generic (PLEG): container finished" podID="4d92c850-d415-4bc7-9fb2-34065e6411de" containerID="e18621a8dead747532c96709ec207a73c06c464303e21760715c79010603d1fa" exitCode=0
Sep 11 18:21:57.998399 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:57.998243    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk"
Sep 11 18:21:57.998399 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:57.998263    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk" event={"ID":"4d92c850-d415-4bc7-9fb2-34065e6411de","Type":"ContainerDied","Data":"e18621a8dead747532c96709ec207a73c06c464303e21760715c79010603d1fa"}
Sep 11 18:21:57.998399 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:57.998380    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk" event={"ID":"4d92c850-d415-4bc7-9fb2-34065e6411de","Type":"ContainerDied","Data":"4f46c91966d5bc57da36b3516d9e876fb6b0d749308cdd37cb638ddb73d64fcc"}
Sep 11 18:21:57.998530 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:57.998413    2559 scope.go:117] "RemoveContainer" containerID="e18621a8dead747532c96709ec207a73c06c464303e21760715c79010603d1fa"
Sep 11 18:21:58.008336 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:58.008311    2559 scope.go:117] "RemoveContainer" containerID="e18621a8dead747532c96709ec207a73c06c464303e21760715c79010603d1fa"
Sep 11 18:21:58.010192 ip-10-0-138-96 kubenswrapper[2559]: E0911 18:21:58.010167    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e18621a8dead747532c96709ec207a73c06c464303e21760715c79010603d1fa\": container with ID starting with e18621a8dead747532c96709ec207a73c06c464303e21760715c79010603d1fa not found: ID does not exist" containerID="e18621a8dead747532c96709ec207a73c06c464303e21760715c79010603d1fa"
Sep 11 18:21:58.010320 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:58.010199    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e18621a8dead747532c96709ec207a73c06c464303e21760715c79010603d1fa"} err="failed to get container status \"e18621a8dead747532c96709ec207a73c06c464303e21760715c79010603d1fa\": rpc error: code = NotFound desc = could not find container \"e18621a8dead747532c96709ec207a73c06c464303e21760715c79010603d1fa\": container with ID starting with e18621a8dead747532c96709ec207a73c06c464303e21760715c79010603d1fa not found: ID does not exist"
Sep 11 18:21:58.027289 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:58.027263    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk"]
Sep 11 18:21:58.031298 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:58.031274    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-55bbb86db8-k74jk"]
Sep 11 18:21:59.052834 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:21:59.052797    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d92c850-d415-4bc7-9fb2-34065e6411de" path="/var/lib/kubelet/pods/4d92c850-d415-4bc7-9fb2-34065e6411de/volumes"
Sep 11 18:22:01.333711 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:01.333677    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:22:08.033157 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:08.033120    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dfgnf"]
Sep 11 18:22:10.782450 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:10.782411    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d6dff65c8-qq67f_088960ae-3c43-479f-b684-0087be132b13/authorino/0.log"
Sep 11 18:22:17.426675 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:17.426597    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f44bff486-v8pcs_f294e044-ccaf-4816-9149-c66d0cb26394/copy-manifests/0.log"
Sep 11 18:22:17.435585 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:17.435562    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f44bff486-v8pcs_f294e044-ccaf-4816-9149-c66d0cb26394/manager/0.log"
Sep 11 18:22:18.093644 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:18.093609    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5c66b56868-mwhzm_224a4296-05d2-4221-a735-aa666ac1a846/maas-api/0.log"
Sep 11 18:22:18.216165 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:18.216138    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819160-db8gv_22904b5e-730e-4d2f-b488-1f48cb9e2052/cleanup/0.log"
Sep 11 18:22:19.300990 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:19.300959    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d6dff65c8-qq67f_088960ae-3c43-479f-b684-0087be132b13/authorino/0.log"
Sep 11 18:22:19.678072 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:19.678034    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-7jtn6_28fa71d6-fe51-46e4-8664-02f82843e893/kuadrant-console-plugin/0.log"
Sep 11 18:22:19.928550 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:19.928473    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-dfgnf_7eaff0ad-e5ca-4029-99bc-4835f2dd95c1/limitador/0.log"
Sep 11 18:22:20.722671 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:20.722635    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77c49cb9c9-4kwsl_15500cdb-e686-4698-97fb-a0d0c5756a17/kube-auth-proxy/0.log"
Sep 11 18:22:21.396145 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:21.396112    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68f69558bb-s4pcz_15f610bc-3cc9-46de-b61c-bae98f80781f/router/0.log"
Sep 11 18:22:22.278020 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:22.277987    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcs46x_8282f8da-b17f-4872-b70e-391487cbefee/main/0.log"
Sep 11 18:22:22.544660 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:22.544582    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-csbfp_f581ce8c-7f44-478d-8936-37010f86a176/main/0.log"
Sep 11 18:22:32.061474 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:32.061436    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rsgml_8890e4f4-b248-4fe7-b3e7-96fec145ce34/global-pull-secret-syncer/0.log"
Sep 11 18:22:32.203513 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:32.203481    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-p2znc_a65223d8-9c38-454a-a958-120affb54b81/konnectivity-agent/0.log"
Sep 11 18:22:32.286337 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:32.286305    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-96.ec2.internal_aa3822d42868560dd3e2a5d526ebe096/haproxy/0.log"
Sep 11 18:22:37.126595 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:37.126556    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d6dff65c8-qq67f_088960ae-3c43-479f-b684-0087be132b13/authorino/0.log"
Sep 11 18:22:37.228248 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:37.228219    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-7jtn6_28fa71d6-fe51-46e4-8664-02f82843e893/kuadrant-console-plugin/0.log"
Sep 11 18:22:37.310954 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:37.310927    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-dfgnf_7eaff0ad-e5ca-4029-99bc-4835f2dd95c1/limitador/0.log"
Sep 11 18:22:39.013998 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:39.013906    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a/alertmanager/0.log"
Sep 11 18:22:39.042772 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:39.042748    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a/config-reloader/0.log"
Sep 11 18:22:39.073941 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:39.073909    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a/kube-rbac-proxy-web/0.log"
Sep 11 18:22:39.113248 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:39.113223    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a/kube-rbac-proxy/0.log"
Sep 11 18:22:39.140807 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:39.140778    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a/kube-rbac-proxy-metric/0.log"
Sep 11 18:22:39.178723 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:39.178696    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a/prom-label-proxy/0.log"
Sep 11 18:22:39.211977 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:39.211919    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5b4a3f0-77ba-4029-bb4b-6e6e66ddbf4a/init-config-reloader/0.log"
Sep 11 18:22:39.378713 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:39.378614    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5759b79cdb-cn79g_c21db2eb-ad47-4d40-815e-535bea98791a/metrics-server/0.log"
Sep 11 18:22:39.445610 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:39.445577    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6cmmd_4ac803ca-e537-4302-848e-ca84419231b4/node-exporter/0.log"
Sep 11 18:22:39.471933 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:39.471905    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6cmmd_4ac803ca-e537-4302-848e-ca84419231b4/kube-rbac-proxy/0.log"
Sep 11 18:22:39.496972 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:39.496948    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6cmmd_4ac803ca-e537-4302-848e-ca84419231b4/init-textfile/0.log"
Sep 11 18:22:40.165642 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:40.165599    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-fb9td_ad24ed04-4ced-4532-9c50-aefac7647b00/prometheus-operator/0.log"
Sep 11 18:22:40.188996 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:40.188956    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-fb9td_ad24ed04-4ced-4532-9c50-aefac7647b00/kube-rbac-proxy/0.log"
Sep 11 18:22:40.243581 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:40.243553    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5865d99ff4-rswtx_2b834f53-64b2-45b3-95b9-e148f9aef7cf/telemeter-client/0.log"
Sep 11 18:22:40.281827 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:40.281800    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5865d99ff4-rswtx_2b834f53-64b2-45b3-95b9-e148f9aef7cf/reload/0.log"
Sep 11 18:22:40.308108 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:40.308083    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5865d99ff4-rswtx_2b834f53-64b2-45b3-95b9-e148f9aef7cf/kube-rbac-proxy/0.log"
Sep 11 18:22:41.716610 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:41.716579    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-j4nxr_3ca7f28e-65fb-42e3-825a-f3100e244207/networking-console-plugin/0.log"
Sep 11 18:22:42.071336 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.071253    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq"]
Sep 11 18:22:42.071660 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.071645    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d92c850-d415-4bc7-9fb2-34065e6411de" containerName="maas-api"
Sep 11 18:22:42.071660 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.071661    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d92c850-d415-4bc7-9fb2-34065e6411de" containerName="maas-api"
Sep 11 18:22:42.071744 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.071723    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d92c850-d415-4bc7-9fb2-34065e6411de" containerName="maas-api"
Sep 11 18:22:42.075017 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.074994    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq"
Sep 11 18:22:42.077538 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.077516    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9wf2k\"/\"kube-root-ca.crt\""
Sep 11 18:22:42.078233 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.078217    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9wf2k\"/\"default-dockercfg-zgzn2\""
Sep 11 18:22:42.078320 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.078235    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9wf2k\"/\"openshift-service-ca.crt\""
Sep 11 18:22:42.089702 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.089674    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq"]
Sep 11 18:22:42.182043 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.182008    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c08f140-dc2d-42f5-8bcd-76dba49f20c1-sys\") pod \"perf-node-gather-daemonset-l8nkq\" (UID: \"1c08f140-dc2d-42f5-8bcd-76dba49f20c1\") " pod="openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq"
Sep 11 18:22:42.182236 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.182081    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c08f140-dc2d-42f5-8bcd-76dba49f20c1-lib-modules\") pod \"perf-node-gather-daemonset-l8nkq\" (UID: \"1c08f140-dc2d-42f5-8bcd-76dba49f20c1\") " pod="openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq"
Sep 11 18:22:42.182236 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.182201    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1c08f140-dc2d-42f5-8bcd-76dba49f20c1-proc\") pod \"perf-node-gather-daemonset-l8nkq\" (UID: \"1c08f140-dc2d-42f5-8bcd-76dba49f20c1\") " pod="openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq"
Sep 11 18:22:42.182236 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.182230    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lvfg\" (UniqueName: \"kubernetes.io/projected/1c08f140-dc2d-42f5-8bcd-76dba49f20c1-kube-api-access-5lvfg\") pod \"perf-node-gather-daemonset-l8nkq\" (UID: \"1c08f140-dc2d-42f5-8bcd-76dba49f20c1\") " pod="openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq"
Sep 11 18:22:42.182380 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.182273    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1c08f140-dc2d-42f5-8bcd-76dba49f20c1-podres\") pod \"perf-node-gather-daemonset-l8nkq\" (UID: \"1c08f140-dc2d-42f5-8bcd-76dba49f20c1\") " pod="openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq"
Sep 11 18:22:42.282972 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.282941    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c08f140-dc2d-42f5-8bcd-76dba49f20c1-lib-modules\") pod \"perf-node-gather-daemonset-l8nkq\" (UID: \"1c08f140-dc2d-42f5-8bcd-76dba49f20c1\") " pod="openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq"
Sep 11 18:22:42.283144 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.282999    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1c08f140-dc2d-42f5-8bcd-76dba49f20c1-proc\") pod \"perf-node-gather-daemonset-l8nkq\" (UID: \"1c08f140-dc2d-42f5-8bcd-76dba49f20c1\") " pod="openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq"
Sep 11 18:22:42.283144 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.283023    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lvfg\" (UniqueName: \"kubernetes.io/projected/1c08f140-dc2d-42f5-8bcd-76dba49f20c1-kube-api-access-5lvfg\") pod \"perf-node-gather-daemonset-l8nkq\" (UID: \"1c08f140-dc2d-42f5-8bcd-76dba49f20c1\") " pod="openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq"
Sep 11 18:22:42.283144 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.283051    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1c08f140-dc2d-42f5-8bcd-76dba49f20c1-podres\") pod \"perf-node-gather-daemonset-l8nkq\" (UID: \"1c08f140-dc2d-42f5-8bcd-76dba49f20c1\") " pod="openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq"
Sep 11 18:22:42.283144 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.283080    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c08f140-dc2d-42f5-8bcd-76dba49f20c1-sys\") pod \"perf-node-gather-daemonset-l8nkq\" (UID: \"1c08f140-dc2d-42f5-8bcd-76dba49f20c1\") " pod="openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq"
Sep 11 18:22:42.283144 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.283111    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c08f140-dc2d-42f5-8bcd-76dba49f20c1-lib-modules\") pod \"perf-node-gather-daemonset-l8nkq\" (UID: \"1c08f140-dc2d-42f5-8bcd-76dba49f20c1\") " pod="openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq"
Sep 11 18:22:42.283144 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.283133    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1c08f140-dc2d-42f5-8bcd-76dba49f20c1-proc\") pod \"perf-node-gather-daemonset-l8nkq\" (UID: \"1c08f140-dc2d-42f5-8bcd-76dba49f20c1\") " pod="openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq"
Sep 11 18:22:42.283425 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.283151    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c08f140-dc2d-42f5-8bcd-76dba49f20c1-sys\") pod \"perf-node-gather-daemonset-l8nkq\" (UID: \"1c08f140-dc2d-42f5-8bcd-76dba49f20c1\") " pod="openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq"
Sep 11 18:22:42.283425 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.283204    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1c08f140-dc2d-42f5-8bcd-76dba49f20c1-podres\") pod \"perf-node-gather-daemonset-l8nkq\" (UID: \"1c08f140-dc2d-42f5-8bcd-76dba49f20c1\") " pod="openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq"
Sep 11 18:22:42.292872 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.292846    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lvfg\" (UniqueName: \"kubernetes.io/projected/1c08f140-dc2d-42f5-8bcd-76dba49f20c1-kube-api-access-5lvfg\") pod \"perf-node-gather-daemonset-l8nkq\" (UID: \"1c08f140-dc2d-42f5-8bcd-76dba49f20c1\") " pod="openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq"
Sep 11 18:22:42.384861 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.384831    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq"
Sep 11 18:22:42.660937 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:42.660908    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq"]
Sep 11 18:22:42.662798 ip-10-0-138-96 kubenswrapper[2559]: W0911 18:22:42.662766    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1c08f140_dc2d_42f5_8bcd_76dba49f20c1.slice/crio-1f7a983faa4d0fed0d2fa88ffe34452b2da47d017f635a87502ebfe6a7526ba2 WatchSource:0}: Error finding container 1f7a983faa4d0fed0d2fa88ffe34452b2da47d017f635a87502ebfe6a7526ba2: Status 404 returned error can't find the container with id 1f7a983faa4d0fed0d2fa88ffe34452b2da47d017f635a87502ebfe6a7526ba2
Sep 11 18:22:43.025831 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:43.025749    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-5d2qv_4546e901-4e48-42bd-860c-54b99428070a/download-server/0.log"
Sep 11 18:22:43.237755 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:43.237719    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq" event={"ID":"1c08f140-dc2d-42f5-8bcd-76dba49f20c1","Type":"ContainerStarted","Data":"fcd322863cd2b72d4383e266bde9d8a2eb0fc596c72ca73c3ab7805b766edf66"}
Sep 11 18:22:43.237958 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:43.237774    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq" event={"ID":"1c08f140-dc2d-42f5-8bcd-76dba49f20c1","Type":"ContainerStarted","Data":"1f7a983faa4d0fed0d2fa88ffe34452b2da47d017f635a87502ebfe6a7526ba2"}
Sep 11 18:22:43.237958 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:43.237853    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq"
Sep 11 18:22:43.257306 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:43.257256    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq" podStartSLOduration=1.257240322 podStartE2EDuration="1.257240322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:22:43.256047174 +0000 UTC m=+2268.701147746" watchObservedRunningTime="2026-09-11 18:22:43.257240322 +0000 UTC m=+2268.702340893"
Sep 11 18:22:44.892299 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:44.892247    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h8grx_e831333a-4c9b-4a40-a53e-c66f3e2f653f/dns/0.log"
Sep 11 18:22:44.916956 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:44.916930    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h8grx_e831333a-4c9b-4a40-a53e-c66f3e2f653f/kube-rbac-proxy/0.log"
Sep 11 18:22:44.999063 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:44.999033    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5qftl_190c54c5-ae20-4ef1-b9b2-af966c1712f4/dns-node-resolver/0.log"
Sep 11 18:22:45.607305 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:45.607267    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-59cc77dfdd-2vlmn_d650e928-1422-4436-8122-308bcb12e93b/registry/0.log"
Sep 11 18:22:45.659396 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:45.659344    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kpjjv_a8238ce9-9f28-4900-8b41-667fa71978f2/node-ca/0.log"
Sep 11 18:22:47.252854 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:47.252819    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77c49cb9c9-4kwsl_15500cdb-e686-4698-97fb-a0d0c5756a17/kube-auth-proxy/0.log"
Sep 11 18:22:47.502020 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:47.501989    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68f69558bb-s4pcz_15f610bc-3cc9-46de-b61c-bae98f80781f/router/0.log"
Sep 11 18:22:48.392437 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:48.392400    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cspnd_37764305-549b-498a-967e-c24cbc84e019/serve-healthcheck-canary/0.log"
Sep 11 18:22:48.919919 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:48.919880    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-lcltf_c539f9b8-d329-47d0-a8ca-79ad211a80c3/insights-operator/0.log"
Sep 11 18:22:48.921208 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:48.921167    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-lcltf_c539f9b8-d329-47d0-a8ca-79ad211a80c3/insights-operator/1.log"
Sep 11 18:22:49.044893 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:49.044861    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qx9t7_950b3a2d-acab-436e-85c1-2ebf115f516b/kube-rbac-proxy/0.log"
Sep 11 18:22:49.072639 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:49.072610    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qx9t7_950b3a2d-acab-436e-85c1-2ebf115f516b/exporter/0.log"
Sep 11 18:22:49.099987 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:49.099953    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qx9t7_950b3a2d-acab-436e-85c1-2ebf115f516b/extractor/0.log"
Sep 11 18:22:49.254159 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:49.254084    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9wf2k/perf-node-gather-daemonset-l8nkq"
Sep 11 18:22:51.960064 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:51.960034    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f44bff486-v8pcs_f294e044-ccaf-4816-9149-c66d0cb26394/manager/0.log"
Sep 11 18:22:51.985977 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:51.985953    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f44bff486-v8pcs_f294e044-ccaf-4816-9149-c66d0cb26394/copy-manifests/0.log"
Sep 11 18:22:53.252272 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:53.252239    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77c49cb9c9-4kwsl_15500cdb-e686-4698-97fb-a0d0c5756a17/kube-auth-proxy/0.log"
Sep 11 18:22:53.392925 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:53.392874    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68f69558bb-s4pcz_15f610bc-3cc9-46de-b61c-bae98f80781f/router/0.log"
Sep 11 18:22:53.591529 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:53.591498    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-f67fb7585-lmq4h_124e9246-3163-48ed-a3e5-a55fc414b38a/manager/0.log"
Sep 11 18:22:55.505902 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:55.505875    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77c49cb9c9-4kwsl_15500cdb-e686-4698-97fb-a0d0c5756a17/kube-auth-proxy/0.log"
Sep 11 18:22:55.653682 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:55.653652    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68f69558bb-s4pcz_15f610bc-3cc9-46de-b61c-bae98f80781f/router/0.log"
Sep 11 18:22:59.053614 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:59.053585    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-4hw4h_abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2/kube-storage-version-migrator-operator/1.log"
Sep 11 18:22:59.054645 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:22:59.054625    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-4hw4h_abe4e709-ba73-4a3a-9f8c-4e6b68bf25f2/kube-storage-version-migrator-operator/0.log"
Sep 11 18:23:00.337412 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:23:00.337337    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d9qt2_d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8/kube-multus-additional-cni-plugins/0.log"
Sep 11 18:23:00.363141 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:23:00.363114    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d9qt2_d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8/egress-router-binary-copy/0.log"
Sep 11 18:23:00.392178 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:23:00.392154    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d9qt2_d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8/cni-plugins/0.log"
Sep 11 18:23:00.422113 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:23:00.422090    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d9qt2_d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8/bond-cni-plugin/0.log"
Sep 11 18:23:00.444722 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:23:00.444697    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d9qt2_d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8/routeoverride-cni/0.log"
Sep 11 18:23:00.470185 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:23:00.470162    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d9qt2_d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8/whereabouts-cni-bincopy/0.log"
Sep 11 18:23:00.494586 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:23:00.494554    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d9qt2_d6f2f35a-7ce1-4f08-9ef1-2e9b481141e8/whereabouts-cni/0.log"
Sep 11 18:23:00.779656 ip-10-0-138-96 kubenswrapper[2559]: I0911 18:23:00.779626    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wh2s7_7d33fe75-e2cd-43e9-9ab1-0078a32c48cc/kube-multus/0.log"