Sep 17 00:07:44.405179 ip-10-0-139-27 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 00:07:44.405332 ip-10-0-139-27 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 00:07:44.405428 ip-10-0-139-27 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 00:07:44.405734 ip-10-0-139-27 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 00:07:54.619907 ip-10-0-139-27 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 00:07:54.619928 ip-10-0-139-27 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 8f3ff47680f2435ca12f848768979119 --
Sep 17 00:09:47.026734 ip-10-0-139-27 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 00:09:47.417702 ip-10-0-139-27 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 00:09:47.417702 ip-10-0-139-27 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 00:09:47.417702 ip-10-0-139-27 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 00:09:47.417702 ip-10-0-139-27 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 00:09:47.417702 ip-10-0-139-27 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 00:09:47.418787 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.418693    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 00:09:47.422973 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.422957    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 00:09:47.422973 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.422973    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 00:09:47.423046 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.422977    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 00:09:47.423046 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.422980    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 00:09:47.423046 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.422983    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 00:09:47.423046 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.422986    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 00:09:47.423046 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.422990    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 00:09:47.423046 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.422993    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 00:09:47.423046 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.422996    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 00:09:47.423046 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.422999    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 00:09:47.423046 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423002    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 00:09:47.423046 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423005    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 00:09:47.423046 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423007    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 00:09:47.423046 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423010    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 00:09:47.423046 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423018    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 00:09:47.423046 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423021    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 00:09:47.423046 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423024    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 00:09:47.423046 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423027    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 00:09:47.423046 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423029    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 00:09:47.423046 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423032    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 00:09:47.423046 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423035    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 00:09:47.423519 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423037    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 00:09:47.423519 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423041    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 00:09:47.423519 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423043    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 00:09:47.423519 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423048    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 00:09:47.423519 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423052    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 00:09:47.423519 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423055    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 00:09:47.423519 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423058    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 00:09:47.423519 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423062    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 00:09:47.423519 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423065    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 00:09:47.423519 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423068    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 00:09:47.423519 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423071    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 00:09:47.423519 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423073    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 00:09:47.423519 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423076    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 00:09:47.423519 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423079    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 00:09:47.423519 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423081    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 00:09:47.423519 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423084    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 00:09:47.423519 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423087    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 00:09:47.423519 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423089    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 00:09:47.423519 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423092    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 00:09:47.423519 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423094    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 00:09:47.424044 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423097    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 00:09:47.424044 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423099    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 00:09:47.424044 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423102    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 00:09:47.424044 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423104    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 00:09:47.424044 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423107    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 00:09:47.424044 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423110    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 00:09:47.424044 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423113    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 00:09:47.424044 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423115    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 00:09:47.424044 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423118    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 00:09:47.424044 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423120    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 00:09:47.424044 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423123    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 00:09:47.424044 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423126    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 00:09:47.424044 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423128    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 00:09:47.424044 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423132    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 00:09:47.424044 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423136    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 00:09:47.424044 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423138    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 00:09:47.424044 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423141    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 00:09:47.424044 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423144    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 00:09:47.424044 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423147    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 00:09:47.424044 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423149    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 00:09:47.424547 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423152    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 00:09:47.424547 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423154    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 00:09:47.424547 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423157    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 00:09:47.424547 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423159    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 00:09:47.424547 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423163    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 00:09:47.424547 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423167    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 00:09:47.424547 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423170    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 00:09:47.424547 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423173    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 00:09:47.424547 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423175    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 00:09:47.424547 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423178    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 00:09:47.424547 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423181    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 00:09:47.424547 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423185    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 00:09:47.424547 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423188    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 00:09:47.424547 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423191    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 00:09:47.424547 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423194    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 00:09:47.424547 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423198    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 00:09:47.424547 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423201    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 00:09:47.424547 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423205    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 00:09:47.424547 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423209    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 00:09:47.425018 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423212    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 00:09:47.425018 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423214    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 00:09:47.425018 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423217    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 00:09:47.425018 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423219    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 00:09:47.425018 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423222    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 00:09:47.425018 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423224    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 00:09:47.425018 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423229    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 00:09:47.425018 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423232    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 00:09:47.425018 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423235    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 00:09:47.425018 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423238    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 00:09:47.425018 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423241    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 00:09:47.425018 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423243    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 00:09:47.425018 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423246    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 00:09:47.425018 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423248    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 00:09:47.425018 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423251    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 00:09:47.425018 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423254    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 00:09:47.425018 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423256    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 00:09:47.425018 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423259    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 00:09:47.425018 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.423262    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 00:09:47.425018 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424008    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 00:09:47.425526 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424016    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 00:09:47.425526 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424019    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 00:09:47.425526 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424023    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 00:09:47.425526 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424026    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 00:09:47.425526 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424029    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 00:09:47.425526 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424033    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 00:09:47.425526 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424035    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 00:09:47.425526 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424038    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 00:09:47.425526 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424041    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 00:09:47.425526 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424044    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 00:09:47.425526 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424048    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 00:09:47.425526 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424051    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 00:09:47.425526 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424054    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 00:09:47.425526 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424057    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 00:09:47.425526 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424061    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 00:09:47.425526 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424063    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 00:09:47.425526 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424066    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 00:09:47.425526 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424069    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 00:09:47.425526 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424073    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 00:09:47.425997 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424077    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 00:09:47.425997 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424080    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 00:09:47.425997 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424083    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 00:09:47.425997 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424086    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 00:09:47.425997 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424089    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 00:09:47.425997 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424092    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 00:09:47.425997 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424095    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 00:09:47.425997 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424098    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 00:09:47.425997 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424100    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 00:09:47.425997 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424103    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 00:09:47.425997 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424106    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 00:09:47.425997 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424109    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 00:09:47.425997 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424112    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 00:09:47.425997 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424114    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 00:09:47.425997 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424117    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 00:09:47.425997 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424120    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 00:09:47.425997 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424122    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 00:09:47.425997 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424125    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 00:09:47.425997 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424129    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 00:09:47.426568 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424132    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 00:09:47.426568 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424135    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 00:09:47.426568 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424137    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 00:09:47.426568 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424140    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 00:09:47.426568 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424143    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 00:09:47.426568 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424146    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 00:09:47.426568 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424148    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 00:09:47.426568 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424152    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 00:09:47.426568 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424155    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 00:09:47.426568 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424159    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 00:09:47.426568 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424162    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 00:09:47.426568 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424165    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 00:09:47.426568 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424168    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 00:09:47.426568 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424171    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 00:09:47.426568 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424174    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 00:09:47.426568 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424177    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 00:09:47.426568 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424179    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 00:09:47.426568 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424182    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 00:09:47.426568 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424185    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 00:09:47.426568 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424187    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 00:09:47.427061 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424190    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 00:09:47.427061 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424193    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 00:09:47.427061 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424195    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 00:09:47.427061 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424198    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 00:09:47.427061 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424201    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 00:09:47.427061 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424203    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 00:09:47.427061 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424206    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 00:09:47.427061 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424208    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 00:09:47.427061 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424211    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 00:09:47.427061 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424213    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 00:09:47.427061 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424216    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 00:09:47.427061 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424218    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 00:09:47.427061 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424221    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 00:09:47.427061 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424224    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 00:09:47.427061 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424226    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 00:09:47.427061 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424229    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 00:09:47.427061 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424232    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 00:09:47.427061 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424234    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 00:09:47.427061 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424238    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 00:09:47.427061 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424240    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 00:09:47.427595 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424245    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 00:09:47.427595 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424248    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 00:09:47.427595 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424250    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 00:09:47.427595 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424253    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 00:09:47.427595 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424255    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 00:09:47.427595 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424258    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 00:09:47.427595 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424261    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 00:09:47.427595 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424264    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 00:09:47.427595 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424267    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 00:09:47.427595 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424270    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 00:09:47.427595 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424272    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 00:09:47.427595 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424275    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 00:09:47.427595 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424277    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 00:09:47.427595 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424280    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 00:09:47.427595 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424283    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 00:09:47.427595 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424286    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 00:09:47.427595 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424288    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 00:09:47.427595 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424291    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 00:09:47.427595 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424293    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 00:09:47.427595 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424297    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 00:09:47.428098 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424698    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 00:09:47.428098 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424703    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 00:09:47.428098 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424706    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 00:09:47.428098 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424709    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 00:09:47.428098 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424711    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 00:09:47.428098 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424714    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 00:09:47.428098 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424717    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 00:09:47.428098 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424719    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 00:09:47.428098 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424722    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 00:09:47.428098 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424725    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 00:09:47.428098 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424727    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 00:09:47.428098 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424730    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 00:09:47.428098 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424733    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 00:09:47.428098 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424736    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 00:09:47.428098 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424738    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 00:09:47.428098 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424742    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 00:09:47.428098 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424745    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 00:09:47.428098 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424748    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 00:09:47.428098 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424751    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 00:09:47.428098 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424753    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 00:09:47.428598 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424757    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 00:09:47.428598 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424759    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 00:09:47.428598 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424762    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 00:09:47.428598 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424765    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 00:09:47.428598 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424769    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 00:09:47.428598 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424773    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 00:09:47.428598 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424776    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 00:09:47.428598 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424781    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 00:09:47.428598 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424784    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 00:09:47.428598 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424787    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 00:09:47.428598 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424790    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 00:09:47.428598 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424792    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 00:09:47.428598 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424795    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 00:09:47.428598 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424798    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 00:09:47.428598 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424800    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 00:09:47.428598 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424802    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 00:09:47.428598 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424805    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 00:09:47.428598 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424808    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 00:09:47.428598 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424810    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 00:09:47.429100 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424813    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 00:09:47.429100 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424815    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 00:09:47.429100 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424818    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 00:09:47.429100 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424820    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 00:09:47.429100 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424823    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 00:09:47.429100 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424825    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 00:09:47.429100 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424828    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 00:09:47.429100 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424831    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 00:09:47.429100 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424836    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 00:09:47.429100 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424839    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 00:09:47.429100 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424842    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 00:09:47.429100 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424844    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 00:09:47.429100 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424847    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 00:09:47.429100 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424849    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 00:09:47.429100 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424852    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 00:09:47.429100 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424855    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 00:09:47.429100 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424858    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 00:09:47.429100 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424860    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 00:09:47.429100 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424863    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 00:09:47.429100 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424865    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 00:09:47.429608 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424868    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 00:09:47.429608 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424870    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 00:09:47.429608 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424873    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 00:09:47.429608 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424876    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 00:09:47.429608 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424879    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 00:09:47.429608 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424881    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 00:09:47.429608 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424884    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 00:09:47.429608 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424887    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 00:09:47.429608 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424890    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 00:09:47.429608 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424892    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 00:09:47.429608 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424895    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 00:09:47.429608 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424897    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 00:09:47.429608 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424900    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 00:09:47.429608 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424903    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 00:09:47.429608 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424905    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 00:09:47.429608 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424908    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 00:09:47.429608 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424910    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 00:09:47.429608 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424913    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 00:09:47.429608 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424915    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 00:09:47.429608 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424919    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 00:09:47.430095 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424923    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 00:09:47.430095 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424926    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 00:09:47.430095 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424928    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 00:09:47.430095 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424931    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 00:09:47.430095 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424934    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 00:09:47.430095 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424937    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 00:09:47.430095 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424939    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 00:09:47.430095 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424942    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 00:09:47.430095 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424946    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 00:09:47.430095 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424950    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 00:09:47.430095 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424953    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 00:09:47.430095 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424956    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 00:09:47.430095 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424958    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 00:09:47.430095 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424961    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 00:09:47.430095 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424964    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 00:09:47.430095 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424967    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 00:09:47.430095 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424969    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 00:09:47.430095 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424980    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 00:09:47.430095 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424983    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 00:09:47.430619 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.424986    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 00:09:47.430619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425062    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 00:09:47.430619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425097    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 00:09:47.430619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425104    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 00:09:47.430619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425109    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 00:09:47.430619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425114    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 00:09:47.430619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425117    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 00:09:47.430619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425122    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 00:09:47.430619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425126    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 00:09:47.430619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425130    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 00:09:47.430619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425133    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 00:09:47.430619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425137    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 00:09:47.430619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425140    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 00:09:47.430619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425145    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 00:09:47.430619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425149    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 17 00:09:47.430619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425152    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 00:09:47.430619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425155    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 17 00:09:47.430619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425158    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 00:09:47.430619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425161    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 00:09:47.430619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425166    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 17 00:09:47.430619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425168    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 00:09:47.430619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425171    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 00:09:47.430619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425175    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 00:09:47.430619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425178    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425182    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425185    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425188    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425191    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425195    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425197    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425200    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425204    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425207    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425211    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425214    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425217    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425220    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425226    2577 flags.go:64] FLAG: --enable-server="true"
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425229    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425234    2577 flags.go:64] FLAG: --event-burst="100"
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425237    2577 flags.go:64] FLAG: --event-qps="50"
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425240    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425243    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425246    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425250    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425254    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425258    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425261    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 17 00:09:47.431201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425264    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 00:09:47.431831 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425267    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 00:09:47.431831 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425270    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 00:09:47.431831 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425273    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 00:09:47.431831 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425276    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 00:09:47.431831 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425279    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 00:09:47.431831 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425282    2577 flags.go:64] FLAG: --feature-gates=""
Sep 17 00:09:47.431831 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425286    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 00:09:47.431831 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425289    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 00:09:47.431831 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425292    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 00:09:47.431831 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425295    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 00:09:47.431831 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425298    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 00:09:47.431831 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425301    2577 flags.go:64] FLAG: --help="false"
Sep 17 00:09:47.431831 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425305    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-139-27.ec2.internal"
Sep 17 00:09:47.431831 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425308    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 00:09:47.431831 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425311    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 00:09:47.431831 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425314    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 00:09:47.431831 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425317    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 00:09:47.431831 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425321    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 00:09:47.431831 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425324    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 00:09:47.431831 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425327    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 00:09:47.431831 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425331    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 00:09:47.431831 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425334    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 00:09:47.431831 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425337    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425340    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425343    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425346    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425349    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425352    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425356    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425359    2577 flags.go:64] FLAG: --lock-file=""
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425362    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425365    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425369    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425374    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425377    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425380    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425382    2577 flags.go:64] FLAG: --logging-format="text"
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425385    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425389    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425392    2577 flags.go:64] FLAG: --manifest-url=""
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425395    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425400    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425403    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425407    2577 flags.go:64] FLAG: --max-pods="110"
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425422    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425426    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425429    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 00:09:47.432436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425432    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425435    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425438    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425441    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425448    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425453    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425456    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425459    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425463    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425469    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425472    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425476    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425479    2577 flags.go:64] FLAG: --port="10250"
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425482    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425486    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-006e9338b02650dc8"
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425489    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425493    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425496    2577 flags.go:64] FLAG: --register-node="true"
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425499    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425503    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425506    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425509    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425511    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425515    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425519    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 00:09:47.433311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425522    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425525    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425528    2577 flags.go:64] FLAG: --runonce="false"
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425531    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425534    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425537    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425540    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425543    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425546    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425550    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425553    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425556    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425560    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425563    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425566    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425569    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425572    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425575    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425580    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425583    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425586    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425592    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425594    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425597    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425601    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 00:09:47.434174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425603    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 00:09:47.434865 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425606    2577 flags.go:64] FLAG: --v="2"
Sep 17 00:09:47.434865 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425610    2577 flags.go:64] FLAG: --version="false"
Sep 17 00:09:47.434865 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425614    2577 flags.go:64] FLAG: --vmodule=""
Sep 17 00:09:47.434865 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425623    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 00:09:47.434865 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.425626    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 00:09:47.434865 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425742    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 00:09:47.434865 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425745    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 00:09:47.434865 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425748    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 00:09:47.434865 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425751    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 00:09:47.434865 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425754    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 00:09:47.434865 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425756    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 00:09:47.434865 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425759    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 00:09:47.434865 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425762    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 00:09:47.434865 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425764    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 00:09:47.434865 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425767    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 00:09:47.434865 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425769    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 00:09:47.434865 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425772    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 00:09:47.434865 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425775    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 00:09:47.434865 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425781    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 00:09:47.434865 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425784    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 00:09:47.434865 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425786    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 00:09:47.435386 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425790    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 00:09:47.435386 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425794    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 00:09:47.435386 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425798    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 00:09:47.435386 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425802    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 00:09:47.435386 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425805    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 00:09:47.435386 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425807    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 00:09:47.435386 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425810    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 00:09:47.435386 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425813    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 00:09:47.435386 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425816    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 00:09:47.435386 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425819    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 00:09:47.435386 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425822    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 00:09:47.435386 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425825    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 00:09:47.435386 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425828    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 00:09:47.435386 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425831    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 00:09:47.435386 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425833    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 00:09:47.435386 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425836    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 00:09:47.435386 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425839    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 00:09:47.435386 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425841    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 00:09:47.435386 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425844    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 00:09:47.435386 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425846    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 00:09:47.435929 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425849    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 00:09:47.435929 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425852    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 00:09:47.435929 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425855    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 00:09:47.435929 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425857    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 00:09:47.435929 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425860    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 00:09:47.435929 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425862    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 00:09:47.435929 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425865    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 00:09:47.435929 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425867    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 00:09:47.435929 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425870    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 00:09:47.435929 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425874    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 00:09:47.435929 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425878    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 00:09:47.435929 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425882    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 00:09:47.435929 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425885    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 00:09:47.435929 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425888    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 00:09:47.435929 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425891    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 00:09:47.435929 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425894    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 00:09:47.435929 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425897    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 00:09:47.435929 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425899    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 00:09:47.435929 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425902    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 00:09:47.436403 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425905    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 00:09:47.436403 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425907    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 00:09:47.436403 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425910    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 00:09:47.436403 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425913    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 00:09:47.436403 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425916    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 00:09:47.436403 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425918    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 00:09:47.436403 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425921    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 00:09:47.436403 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425924    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 00:09:47.436403 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425926    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 00:09:47.436403 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425929    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 00:09:47.436403 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425932    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 00:09:47.436403 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425934    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 00:09:47.436403 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425937    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 00:09:47.436403 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425940    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 00:09:47.436403 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425942    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 00:09:47.436403 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425945    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 00:09:47.436403 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425947    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 00:09:47.436403 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425950    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 00:09:47.436403 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425953    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 00:09:47.436403 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425955    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 00:09:47.436927 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425958    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 00:09:47.436927 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425960    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 00:09:47.436927 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425964    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 00:09:47.436927 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425967    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 00:09:47.436927 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425969    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 00:09:47.436927 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425972    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 00:09:47.436927 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425975    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 00:09:47.436927 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425977    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 00:09:47.436927 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425981    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 00:09:47.436927 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425984    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 00:09:47.436927 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425987    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 00:09:47.436927 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425990    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 00:09:47.436927 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425993    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 00:09:47.436927 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425996    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 00:09:47.436927 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.425999    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 00:09:47.436927 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.426001    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 00:09:47.436927 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.426004    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 00:09:47.436927 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.426007    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 00:09:47.436927 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.426010    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 00:09:47.436927 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.426013    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 00:09:47.437445 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.426016    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 00:09:47.437445 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.426018    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 00:09:47.437445 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.426021    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 00:09:47.437445 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.426025    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 00:09:47.437445 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.426615    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 00:09:47.437445 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.435128    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 00:09:47.437445 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.435143    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 00:09:47.437445 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.435945    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 00:09:47.437445 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.435957    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 00:09:47.437445 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436022    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 00:09:47.437445 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436028    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 00:09:47.437445 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436034    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 00:09:47.437445 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436037    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 00:09:47.437445 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436040    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 00:09:47.437802 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436043    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 00:09:47.437802 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436061    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 00:09:47.437802 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436065    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 00:09:47.437802 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436069    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 00:09:47.437802 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436073    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 00:09:47.437802 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436076    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 00:09:47.437802 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436079    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 00:09:47.437802 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436082    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 00:09:47.437802 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436084    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 00:09:47.437802 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436088    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 00:09:47.437802 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436091    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 00:09:47.437802 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436094    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 00:09:47.437802 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436097    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 00:09:47.437802 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436100    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 00:09:47.437802 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436102    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 00:09:47.437802 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436105    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 00:09:47.437802 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436108    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 00:09:47.437802 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436111    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 00:09:47.437802 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436114    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 00:09:47.437802 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436117    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 00:09:47.438318 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436119    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 00:09:47.438318 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436122    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 00:09:47.438318 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436125    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 00:09:47.438318 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436127    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 00:09:47.438318 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436130    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 00:09:47.438318 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436133    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 00:09:47.438318 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436136    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 00:09:47.438318 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436138    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 00:09:47.438318 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436141    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 00:09:47.438318 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436144    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 00:09:47.438318 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436146    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 00:09:47.438318 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436149    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 00:09:47.438318 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436152    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 00:09:47.438318 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436155    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 00:09:47.438318 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436158    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 00:09:47.438318 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436160    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 00:09:47.438318 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436163    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 00:09:47.438318 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436166    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 00:09:47.438318 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436169    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 00:09:47.438807 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436171    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 00:09:47.438807 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436174    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 00:09:47.438807 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436177    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 00:09:47.438807 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436180    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 00:09:47.438807 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436183    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 00:09:47.438807 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436187    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 00:09:47.438807 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436189    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 00:09:47.438807 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436192    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 00:09:47.438807 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436195    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 00:09:47.438807 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436198    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 00:09:47.438807 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436200    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 00:09:47.438807 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436203    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 00:09:47.438807 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436205    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 00:09:47.438807 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436208    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 00:09:47.438807 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436211    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 00:09:47.438807 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436213    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 00:09:47.438807 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436216    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 00:09:47.438807 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436218    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 00:09:47.438807 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436221    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 00:09:47.438807 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436224    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 00:09:47.439305 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436227    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 00:09:47.439305 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436229    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 00:09:47.439305 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436232    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 00:09:47.439305 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436235    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 00:09:47.439305 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436237    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 00:09:47.439305 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436240    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 00:09:47.439305 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436243    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 00:09:47.439305 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436245    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 00:09:47.439305 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436248    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 00:09:47.439305 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436250    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 00:09:47.439305 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436253    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 00:09:47.439305 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436256    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 00:09:47.439305 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436260    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 00:09:47.439305 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436263    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 00:09:47.439305 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436267    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 00:09:47.439305 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436271    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 00:09:47.439305 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436273    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 00:09:47.439305 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436276    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 00:09:47.439305 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436279    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 00:09:47.439305 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436282    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 00:09:47.439820 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436285    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 00:09:47.439820 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436287    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 00:09:47.439820 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436290    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 00:09:47.439820 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436292    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 00:09:47.439820 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436295    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 00:09:47.439820 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436298    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 00:09:47.439820 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436302    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 00:09:47.439820 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436305    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 00:09:47.439820 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436308    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 00:09:47.439820 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436311    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 00:09:47.439820 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436314    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 00:09:47.439820 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436317    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 00:09:47.439820 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436319    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 00:09:47.439820 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436322    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 00:09:47.439820 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436324    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 00:09:47.440226 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.436329    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 00:09:47.440226 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436460    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 00:09:47.440226 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436466    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 00:09:47.440226 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436471    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 00:09:47.440226 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436474    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 00:09:47.440226 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436477    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 00:09:47.440226 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436480    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 00:09:47.440226 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436483    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 00:09:47.440226 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436486    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 00:09:47.440226 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436489    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 00:09:47.440226 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436492    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 00:09:47.440226 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436495    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 00:09:47.440226 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436498    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 00:09:47.440226 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436502    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 00:09:47.440226 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436505    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 00:09:47.440632 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436507    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 00:09:47.440632 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436510    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 00:09:47.440632 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436512    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 00:09:47.440632 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436517    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 00:09:47.440632 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436521    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 00:09:47.440632 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436524    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 00:09:47.440632 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436526    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 00:09:47.440632 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436529    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 00:09:47.440632 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436531    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 00:09:47.440632 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436534    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 00:09:47.440632 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436537    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 00:09:47.440632 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436540    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 00:09:47.440632 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436543    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 00:09:47.440632 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436545    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 00:09:47.440632 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436548    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 00:09:47.440632 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436550    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 00:09:47.440632 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436553    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 00:09:47.440632 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436556    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 00:09:47.440632 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436558    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 00:09:47.441124 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436561    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 00:09:47.441124 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436563    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 00:09:47.441124 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436566    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 00:09:47.441124 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436569    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 00:09:47.441124 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436571    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 00:09:47.441124 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436574    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 00:09:47.441124 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436576    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 00:09:47.441124 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436579    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 00:09:47.441124 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436582    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 00:09:47.441124 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436584    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 00:09:47.441124 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436588    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 00:09:47.441124 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436592    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 00:09:47.441124 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436596    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 00:09:47.441124 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436599    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 00:09:47.441124 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436601    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 00:09:47.441124 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436605    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 00:09:47.441124 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436608    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 00:09:47.441124 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436612    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 00:09:47.441124 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436615    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 00:09:47.441616 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436617    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 00:09:47.441616 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436620    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 00:09:47.441616 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436623    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 00:09:47.441616 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436626    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 00:09:47.441616 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436629    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 00:09:47.441616 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436632    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 00:09:47.441616 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436635    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 00:09:47.441616 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436638    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 00:09:47.441616 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436641    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 00:09:47.441616 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436643    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 00:09:47.441616 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436646    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 00:09:47.441616 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436648    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 00:09:47.441616 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436651    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 00:09:47.441616 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436654    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 00:09:47.441616 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436656    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 00:09:47.441616 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436659    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 00:09:47.441616 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436662    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 00:09:47.441616 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436664    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 00:09:47.441616 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436667    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 00:09:47.441616 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436669    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 00:09:47.442106 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436672    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 00:09:47.442106 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436675    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 00:09:47.442106 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436677    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 00:09:47.442106 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436682    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 00:09:47.442106 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436684    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 00:09:47.442106 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436687    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 00:09:47.442106 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436691    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 00:09:47.442106 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436693    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 00:09:47.442106 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436696    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 00:09:47.442106 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436699    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 00:09:47.442106 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436701    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 00:09:47.442106 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436704    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 00:09:47.442106 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436706    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 00:09:47.442106 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436709    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 00:09:47.442106 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436712    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 00:09:47.442106 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436714    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 00:09:47.442106 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436717    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 00:09:47.442106 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436720    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 00:09:47.442106 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436722    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 00:09:47.442106 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436725    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 00:09:47.442685 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436728    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 00:09:47.442685 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436730    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 00:09:47.442685 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436733    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 00:09:47.442685 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436736    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 00:09:47.442685 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436738    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 00:09:47.442685 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436741    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 00:09:47.442685 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:47.436744    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 00:09:47.442685 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.436749    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 00:09:47.442685 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.436864    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 00:09:47.442685 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.439636    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 00:09:47.442685 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.440440    2577 server.go:1019] "Starting client certificate rotation"
Sep 17 00:09:47.442685 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.440549    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 00:09:47.442685 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.440582    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 00:09:47.443031 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.442921    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 00:09:47.444120 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.444101    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 00:09:47.454755 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.454734    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 17 00:09:47.458169 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.458152    2577 log.go:25] "Validated CRI v1 image API"
Sep 17 00:09:47.461497 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.461481    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 00:09:47.465406 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.465383    2577 fs.go:135] Filesystem UUIDs: map[31870690-25a8-4707-9c5e-9c40e2f00a95:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7cf1d283-7113-44db-8c45-4621a2b073cf:/dev/nvme0n1p4]
Sep 17 00:09:47.465497 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.465404    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 00:09:47.472349 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.472231    2577 manager.go:216] Machine: {Timestamp:2026-09-17 00:09:47.471047544 +0000 UTC m=+0.329540670 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098871 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2820f6eab4891177d649b56711e01f SystemUUID:ec2820f6-eab4-8911-77d6-49b56711e01f BootID:8f3ff476-80f2-435c-a12f-848768979119 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d4:00:c9:9b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d4:00:c9:9b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:9b:1b:44:4e:ee Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 00:09:47.472349 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.472341    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 00:09:47.472513 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.472443    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 00:09:47.473428 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.473397    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 00:09:47.475735 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.475708    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 00:09:47.475872 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.475737    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-27.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 00:09:47.476905 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.476895    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 00:09:47.476939 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.476908    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 00:09:47.476939 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.476917    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 00:09:47.477623 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.477613    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 00:09:47.478448 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.478439    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 00:09:47.481199 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.481187    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 00:09:47.481319 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.481306    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 00:09:47.481356 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.481340    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 00:09:47.481356 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.481350    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 00:09:47.481425 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.481362    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 00:09:47.481425 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.481371    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 17 00:09:47.481425 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.481379    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 00:09:47.481739 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.481725    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 00:09:47.481799 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.481745    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 00:09:47.484012 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.483998    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 17 00:09:47.486116 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.486103    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 00:09:47.486161 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.486133    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 00:09:47.487702 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.487690    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 00:09:47.487758 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.487708    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 00:09:47.487758 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.487713    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 00:09:47.487758 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.487719    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 00:09:47.487758 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.487728    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 00:09:47.487758 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.487733    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 00:09:47.487758 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.487747    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 00:09:47.487758 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.487760    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 00:09:47.487939 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.487767    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 00:09:47.487939 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.487773    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 00:09:47.487939 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.487781    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 00:09:47.487939 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.487786    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 00:09:47.489249 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.489234    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 00:09:47.489249 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.489247    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 00:09:47.490892 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:47.490872    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-27.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 00:09:47.490965 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:47.490872    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 00:09:47.493575 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.493555    2577 server.go:1268] "Started kubelet"
Sep 17 00:09:47.493671 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.493620    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 00:09:47.493760 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.493690    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 00:09:47.493850 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.493835    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 00:09:47.494344 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.494323    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-27.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 00:09:47.494816 ip-10-0-139-27 systemd[1]: Started Kubernetes Kubelet.
Sep 17 00:09:47.496042 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.496023    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 00:09:47.497361 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.496567    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 00:09:47.498940 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:47.498918    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 00:09:47.502097 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:47.501126    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-27.ec2.internal.18d5f329ba3e7371  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-27.ec2.internal,UID:ip-10-0-139-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-27.ec2.internal,},FirstTimestamp:2026-09-17 00:09:47.493503857 +0000 UTC m=+0.351997043,LastTimestamp:2026-09-17 00:09:47.493503857 +0000 UTC m=+0.351997043,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-27.ec2.internal,}"
Sep 17 00:09:47.502516 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.502485    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 00:09:47.502856 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.502837    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 00:09:47.503089 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:47.503055    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-27.ec2.internal\" not found"
Sep 17 00:09:47.503612 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.503594    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 00:09:47.503690 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.503623    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 00:09:47.503690 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.503594    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 00:09:47.503769 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.503720    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 00:09:47.503769 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.503732    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 00:09:47.503849 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.503830    2577 factory.go:55] Registering systemd factory
Sep 17 00:09:47.503877 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.503858    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 17 00:09:47.504077 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.504065    2577 factory.go:153] Registering CRI-O factory
Sep 17 00:09:47.504077 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.504078    2577 factory.go:223] Registration of the crio container factory successfully
Sep 17 00:09:47.504142 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.504118    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 00:09:47.504142 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.504132    2577 factory.go:103] Registering Raw factory
Sep 17 00:09:47.504142 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.504142    2577 manager.go:1196] Started watching for new ooms in manager
Sep 17 00:09:47.504516 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.504496    2577 manager.go:349] Starting recovery of all containers
Sep 17 00:09:47.508655 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:47.508627    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 00:09:47.508958 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:47.508928    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-27.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 00:09:47.511679 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.511650    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 00:09:47.517993 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.517806    2577 manager.go:354] Recovery completed
Sep 17 00:09:47.519480 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.519402    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lk2sx"
Sep 17 00:09:47.522568 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.522555    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 00:09:47.525981 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.525963    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-27.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 00:09:47.526066 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.525993    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-27.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 00:09:47.526066 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.526004    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-27.ec2.internal" event="NodeHasSufficientPID"
Sep 17 00:09:47.526296 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.526274    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lk2sx"
Sep 17 00:09:47.526797 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.526783    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 00:09:47.526903 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.526785    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 00:09:47.526903 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.526823    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 00:09:47.526903 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.526840    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 00:09:47.528197 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:47.528112    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-27.ec2.internal.18d5f329bc2e005a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-27.ec2.internal,UID:ip-10-0-139-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-27.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-27.ec2.internal,},FirstTimestamp:2026-09-17 00:09:47.52598025 +0000 UTC m=+0.384473377,LastTimestamp:2026-09-17 00:09:47.52598025 +0000 UTC m=+0.384473377,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-27.ec2.internal,}"
Sep 17 00:09:47.528576 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.528560    2577 policy_none.go:49] "None policy: Start"
Sep 17 00:09:47.528648 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.528597    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 00:09:47.528648 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.528613    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 00:09:47.528758 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.528647    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-27.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 00:09:47.528758 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.528703    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-27.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 00:09:47.528758 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.528716    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-27.ec2.internal" event="NodeHasSufficientPID"
Sep 17 00:09:47.529763 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.529749    2577 policy_none.go:47] "Start"
Sep 17 00:09:47.552143 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.552120    2577 manager.go:337] "Starting Device Plugin manager"
Sep 17 00:09:47.552230 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:47.552181    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 00:09:47.552230 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.552196    2577 server.go:87] "Starting device plugin registration server"
Sep 17 00:09:47.552543 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.552528    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 00:09:47.552642 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.552565    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 00:09:47.552808 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.552790    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 00:09:47.552888 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.552875    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 00:09:47.552952 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.552889    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 00:09:47.553990 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:47.553967    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 00:09:47.554070 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:47.554025    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-27.ec2.internal\" not found"
Sep 17 00:09:47.600846 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.600816    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 00:09:47.601813 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.600867    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 00:09:47.601813 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.600892    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 00:09:47.601813 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:47.600926    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 00:09:47.602471 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.602448    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 00:09:47.653648 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.653624    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 00:09:47.654588 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.654566    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-27.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 00:09:47.654679 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.654602    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-27.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 00:09:47.654679 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.654613    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-27.ec2.internal" event="NodeHasSufficientPID"
Sep 17 00:09:47.654679 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.654635    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-27.ec2.internal"
Sep 17 00:09:47.662904 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.662890    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-27.ec2.internal"
Sep 17 00:09:47.662961 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:47.662912    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-27.ec2.internal\": node \"ip-10-0-139-27.ec2.internal\" not found"
Sep 17 00:09:47.685688 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:47.685611    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-27.ec2.internal\" not found"
Sep 17 00:09:47.701794 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.701773    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-27.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-27.ec2.internal"]
Sep 17 00:09:47.701855 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.701835    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 00:09:47.703347 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.703332    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-27.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 00:09:47.703433 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.703358    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-27.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 00:09:47.703433 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.703368    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-27.ec2.internal" event="NodeHasSufficientPID"
Sep 17 00:09:47.704648 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.704635    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 00:09:47.704767 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.704754    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-27.ec2.internal"
Sep 17 00:09:47.704863 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.704783    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 00:09:47.704863 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.704747    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ff3d9d4f106049a4c07b46e4d77c855d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-27.ec2.internal\" (UID: \"ff3d9d4f106049a4c07b46e4d77c855d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-27.ec2.internal"
Sep 17 00:09:47.704863 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.704825    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff3d9d4f106049a4c07b46e4d77c855d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-27.ec2.internal\" (UID: \"ff3d9d4f106049a4c07b46e4d77c855d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-27.ec2.internal"
Sep 17 00:09:47.705342 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.705323    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-27.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 00:09:47.705446 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.705351    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-27.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 00:09:47.705446 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.705361    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-27.ec2.internal" event="NodeHasSufficientPID"
Sep 17 00:09:47.705446 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.705323    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-27.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 00:09:47.705446 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.705441    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-27.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 00:09:47.705582 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.705457    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-27.ec2.internal" event="NodeHasSufficientPID"
Sep 17 00:09:47.706737 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.706722    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-27.ec2.internal"
Sep 17 00:09:47.706803 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.706745    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 00:09:47.707438 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.707408    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-27.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 00:09:47.707506 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.707447    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-27.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 00:09:47.707506 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.707457    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-27.ec2.internal" event="NodeHasSufficientPID"
Sep 17 00:09:47.731081 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:47.731049    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-27.ec2.internal\" not found" node="ip-10-0-139-27.ec2.internal"
Sep 17 00:09:47.735590 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:47.735573    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-27.ec2.internal\" not found" node="ip-10-0-139-27.ec2.internal"
Sep 17 00:09:47.786627 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:47.786596    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-27.ec2.internal\" not found"
Sep 17 00:09:47.804946 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.804920    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff3d9d4f106049a4c07b46e4d77c855d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-27.ec2.internal\" (UID: \"ff3d9d4f106049a4c07b46e4d77c855d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-27.ec2.internal"
Sep 17 00:09:47.805010 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.804953    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fbf1e8714548f742e80d275d4df6599c-config\") pod \"kube-apiserver-proxy-ip-10-0-139-27.ec2.internal\" (UID: \"fbf1e8714548f742e80d275d4df6599c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-27.ec2.internal"
Sep 17 00:09:47.805010 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.804982    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ff3d9d4f106049a4c07b46e4d77c855d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-27.ec2.internal\" (UID: \"ff3d9d4f106049a4c07b46e4d77c855d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-27.ec2.internal"
Sep 17 00:09:47.805077 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.805043    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff3d9d4f106049a4c07b46e4d77c855d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-27.ec2.internal\" (UID: \"ff3d9d4f106049a4c07b46e4d77c855d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-27.ec2.internal"
Sep 17 00:09:47.805110 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.805050    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ff3d9d4f106049a4c07b46e4d77c855d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-27.ec2.internal\" (UID: \"ff3d9d4f106049a4c07b46e4d77c855d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-27.ec2.internal"
Sep 17 00:09:47.887088 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:47.887049    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-27.ec2.internal\" not found"
Sep 17 00:09:47.905342 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.905321    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fbf1e8714548f742e80d275d4df6599c-config\") pod \"kube-apiserver-proxy-ip-10-0-139-27.ec2.internal\" (UID: \"fbf1e8714548f742e80d275d4df6599c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-27.ec2.internal"
Sep 17 00:09:47.905453 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:47.905365    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fbf1e8714548f742e80d275d4df6599c-config\") pod \"kube-apiserver-proxy-ip-10-0-139-27.ec2.internal\" (UID: \"fbf1e8714548f742e80d275d4df6599c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-27.ec2.internal"
Sep 17 00:09:47.987823 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:47.987751    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-27.ec2.internal\" not found"
Sep 17 00:09:48.034195 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.034170    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-27.ec2.internal"
Sep 17 00:09:48.038902 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.038886    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-27.ec2.internal"
Sep 17 00:09:48.088276 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:48.088240    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-27.ec2.internal\" not found"
Sep 17 00:09:48.188524 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:48.188482    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-27.ec2.internal\" not found"
Sep 17 00:09:48.294907 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:48.294822    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-27.ec2.internal\" not found"
Sep 17 00:09:48.302030 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.302010    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 00:09:48.303477 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.303459    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-27.ec2.internal"
Sep 17 00:09:48.314789 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.314768    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 00:09:48.316687 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.316661    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-27.ec2.internal"
Sep 17 00:09:48.329808 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.329787    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 00:09:48.440958 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.440925    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 00:09:48.441430 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.441075    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 00:09:48.441430 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.441083    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 00:09:48.482538 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.482510    2577 apiserver.go:52] "Watching apiserver"
Sep 17 00:09:48.502620 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.502597    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 00:09:48.502620 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.502613    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 00:09:48.503781 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.503756    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-rg8zr","openshift-multus/multus-additional-cni-plugins-g4cmx","openshift-network-diagnostics/network-check-target-s8lnt","openshift-network-operator/iptables-alerter-4bds9","openshift-ovn-kubernetes/ovnkube-node-bz685","kube-system/global-pull-secret-syncer-ch5w2","kube-system/konnectivity-agent-w8ps8","kube-system/kube-apiserver-proxy-ip-10-0-139-27.ec2.internal","openshift-cluster-node-tuning-operator/tuned-vml9z","openshift-dns/node-resolver-9xxfq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-27.ec2.internal","openshift-multus/multus-k2t7f","openshift-multus/network-metrics-daemon-xbhz5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl"]
Sep 17 00:09:48.505222 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.505198    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-w8ps8"
Sep 17 00:09:48.506440 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.506405    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:09:48.506612 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:48.506488    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbhz5" podUID="b956a62e-c4ac-4e1d-aeb0-3f7907a0a702"
Sep 17 00:09:48.507507 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.507478    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:09:48.507622 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:48.507547    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8lnt" podUID="f5f1a947-783b-4a35-a70b-b850883fef06"
Sep 17 00:09:48.507911 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.507892    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 00:09:48.508009 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.507897    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hvbsm\""
Sep 17 00:09:48.508068 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.508023    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 00:09:48.508761 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.508736    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4bds9"
Sep 17 00:09:48.509982 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.509959    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.511246 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.511220    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:09:48.511342 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:48.511303    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch5w2" podUID="504ae0de-dca4-48aa-964c-b457c02267a5"
Sep 17 00:09:48.511977 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.511957    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 00:09:48.512075 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.512056    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j4phn\""
Sep 17 00:09:48.512358 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.512336    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 00:09:48.512561 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.512541    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 00:09:48.512660 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.512641    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.513056 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.513039    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 00:09:48.513378 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.513363    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nnvdv\""
Sep 17 00:09:48.513742 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.513708    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 00:09:48.513742 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.513733    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 00:09:48.513891 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.513759    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 00:09:48.513891 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.513851    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 00:09:48.513996 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.513918    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 00:09:48.514896 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.514871    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rg8zr"
Sep 17 00:09:48.516095 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.515192    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 00:09:48.516095 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.516071    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-smh58\""
Sep 17 00:09:48.516285 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.516179    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 00:09:48.516866 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.516841    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 00:09:48.518117 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.517447    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g4cmx"
Sep 17 00:09:48.518215 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.518190    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 00:09:48.518275 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.518228    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 00:09:48.518275 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.518235    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 00:09:48.518520 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.518496    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fzlgm\""
Sep 17 00:09:48.519178 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.519153    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9xxfq"
Sep 17 00:09:48.520029 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.520009    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 00:09:48.520315 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.520296    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 00:09:48.520628 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.520604    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 00:09:48.520712 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.520629    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 00:09:48.520712 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.520642    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kk6np\""
Sep 17 00:09:48.520825 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.520814    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 00:09:48.521067 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.521050    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl"
Sep 17 00:09:48.521829 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.521653    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 00:09:48.521829 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.521679    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 00:09:48.521829 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.521749    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vxflv\""
Sep 17 00:09:48.522576 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.522561    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.523593 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.523574    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9c6pv\""
Sep 17 00:09:48.523718 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.523699    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 00:09:48.523849 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.523832    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 00:09:48.524070 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.524057    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 00:09:48.525107 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.525089    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 00:09:48.525201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.525136    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4vvkc\""
Sep 17 00:09:48.529814 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.529770    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 00:04:48 +0000 UTC" deadline="2028-03-29 08:35:41.475593308 +0000 UTC"
Sep 17 00:09:48.529814 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.529814    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13424h25m52.9457828s"
Sep 17 00:09:48.540782 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.540764    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-j79xv"
Sep 17 00:09:48.549660 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.549593    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-j79xv"
Sep 17 00:09:48.604610 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.604585    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 00:09:48.608888 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:48.608860    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3d9d4f106049a4c07b46e4d77c855d.slice/crio-34f75828f2cb62cf5de81863cccdc659cea38cfaee6b9f60351ae512bd718556 WatchSource:0}: Error finding container 34f75828f2cb62cf5de81863cccdc659cea38cfaee6b9f60351ae512bd718556: Status 404 returned error can't find the container with id 34f75828f2cb62cf5de81863cccdc659cea38cfaee6b9f60351ae512bd718556
Sep 17 00:09:48.609001 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.608965    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-sys\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.609059 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.608998    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-lib-modules\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.609059 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609018    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-etc-tuned\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.609059 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609041    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0741efde-4990-4669-9939-d5637d40f0e6-device-dir\") pod \"aws-ebs-csi-driver-node-g5bpl\" (UID: \"0741efde-4990-4669-9939-d5637d40f0e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl"
Sep 17 00:09:48.609202 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0741efde-4990-4669-9939-d5637d40f0e6-etc-selinux\") pod \"aws-ebs-csi-driver-node-g5bpl\" (UID: \"0741efde-4990-4669-9939-d5637d40f0e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl"
Sep 17 00:09:48.609202 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609092    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/504ae0de-dca4-48aa-964c-b457c02267a5-original-pull-secret\") pod \"global-pull-secret-syncer-ch5w2\" (UID: \"504ae0de-dca4-48aa-964c-b457c02267a5\") " pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:09:48.609202 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-systemd-units\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.609202 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609155    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-host-run-ovn-kubernetes\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.609202 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0741efde-4990-4669-9939-d5637d40f0e6-sys-fs\") pod \"aws-ebs-csi-driver-node-g5bpl\" (UID: \"0741efde-4990-4669-9939-d5637d40f0e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl"
Sep 17 00:09:48.609405 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609233    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-host-run-netns\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.609405 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609281    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-multus-conf-dir\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.609405 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609307    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxnfj\" (UniqueName: \"kubernetes.io/projected/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-kube-api-access-bxnfj\") pod \"network-metrics-daemon-xbhz5\" (UID: \"b956a62e-c4ac-4e1d-aeb0-3f7907a0a702\") " pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:09:48.609405 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d261c1cf-76d0-468c-a122-1c2b359f3765-env-overrides\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.609405 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609376    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-var-lib-kubelet\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.609691 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609422    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkpq2\" (UniqueName: \"kubernetes.io/projected/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-kube-api-access-rkpq2\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.609691 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5685bd44-e964-4176-ae6e-479bcee01cbb-iptables-alerter-script\") pod \"iptables-alerter-4bds9\" (UID: \"5685bd44-e964-4176-ae6e-479bcee01cbb\") " pod="openshift-network-operator/iptables-alerter-4bds9"
Sep 17 00:09:48.609691 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609490    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4v9p\" (UniqueName: \"kubernetes.io/projected/5685bd44-e964-4176-ae6e-479bcee01cbb-kube-api-access-n4v9p\") pod \"iptables-alerter-4bds9\" (UID: \"5685bd44-e964-4176-ae6e-479bcee01cbb\") " pod="openshift-network-operator/iptables-alerter-4bds9"
Sep 17 00:09:48.609691 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/504ae0de-dca4-48aa-964c-b457c02267a5-kubelet-config\") pod \"global-pull-secret-syncer-ch5w2\" (UID: \"504ae0de-dca4-48aa-964c-b457c02267a5\") " pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:09:48.609691 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609562    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-run-ovn\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.609691 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0741efde-4990-4669-9939-d5637d40f0e6-socket-dir\") pod \"aws-ebs-csi-driver-node-g5bpl\" (UID: \"0741efde-4990-4669-9939-d5637d40f0e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl"
Sep 17 00:09:48.609691 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhwz4\" (UniqueName: \"kubernetes.io/projected/0a948860-0f14-46ec-afab-e23492108eaa-kube-api-access-zhwz4\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.609691 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609685    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c956d3aa-d437-4b53-afc6-56127f4e1688-konnectivity-ca\") pod \"konnectivity-agent-w8ps8\" (UID: \"c956d3aa-d437-4b53-afc6-56127f4e1688\") " pod="kube-system/konnectivity-agent-w8ps8"
Sep 17 00:09:48.610066 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609716    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-host-slash\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.610066 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609743    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-host-cni-netd\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.610066 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.610066 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d261c1cf-76d0-468c-a122-1c2b359f3765-ovn-node-metrics-cert\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.610066 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609834    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-etc-kubernetes\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.610066 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609857    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-host\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.610066 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609883    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30b22991-ec04-4b85-9b9d-87959cfa30d1-system-cni-dir\") pod \"multus-additional-cni-plugins-g4cmx\" (UID: \"30b22991-ec04-4b85-9b9d-87959cfa30d1\") " pod="openshift-multus/multus-additional-cni-plugins-g4cmx"
Sep 17 00:09:48.610066 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609912    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0741efde-4990-4669-9939-d5637d40f0e6-registration-dir\") pod \"aws-ebs-csi-driver-node-g5bpl\" (UID: \"0741efde-4990-4669-9939-d5637d40f0e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl"
Sep 17 00:09:48.610066 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-host-kubelet\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.610066 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.609977    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/504ae0de-dca4-48aa-964c-b457c02267a5-dbus\") pod \"global-pull-secret-syncer-ch5w2\" (UID: \"504ae0de-dca4-48aa-964c-b457c02267a5\") " pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:09:48.610066 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610001    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-run-openvswitch\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.610066 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-host-cni-bin\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.610066 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610044    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30b22991-ec04-4b85-9b9d-87959cfa30d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-g4cmx\" (UID: \"30b22991-ec04-4b85-9b9d-87959cfa30d1\") " pod="openshift-multus/multus-additional-cni-plugins-g4cmx"
Sep 17 00:09:48.610066 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610073    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30b22991-ec04-4b85-9b9d-87959cfa30d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g4cmx\" (UID: \"30b22991-ec04-4b85-9b9d-87959cfa30d1\") " pod="openshift-multus/multus-additional-cni-plugins-g4cmx"
Sep 17 00:09:48.610775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610099    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0741efde-4990-4669-9939-d5637d40f0e6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g5bpl\" (UID: \"0741efde-4990-4669-9939-d5637d40f0e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl"
Sep 17 00:09:48.610775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610137    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-run\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.610775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30b22991-ec04-4b85-9b9d-87959cfa30d1-cnibin\") pod \"multus-additional-cni-plugins-g4cmx\" (UID: \"30b22991-ec04-4b85-9b9d-87959cfa30d1\") " pod="openshift-multus/multus-additional-cni-plugins-g4cmx"
Sep 17 00:09:48.610775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610189    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e97bfc2-d46a-4384-bc1a-a1a2d37afe75-tmp-dir\") pod \"node-resolver-9xxfq\" (UID: \"3e97bfc2-d46a-4384-bc1a-a1a2d37afe75\") " pod="openshift-dns/node-resolver-9xxfq"
Sep 17 00:09:48.610775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-host-run-k8s-cni-cncf-io\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.610775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-hostroot\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.610775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610366    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-etc-kubernetes\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.610775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610393    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6570468-fe3a-4512-8bda-9880edc3ff96-host\") pod \"node-ca-rg8zr\" (UID: \"e6570468-fe3a-4512-8bda-9880edc3ff96\") " pod="openshift-image-registry/node-ca-rg8zr"
Sep 17 00:09:48.610775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610431    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5685bd44-e964-4176-ae6e-479bcee01cbb-host-slash\") pod \"iptables-alerter-4bds9\" (UID: \"5685bd44-e964-4176-ae6e-479bcee01cbb\") " pod="openshift-network-operator/iptables-alerter-4bds9"
Sep 17 00:09:48.610775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-etc-openvswitch\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.610775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-log-socket\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.610775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30b22991-ec04-4b85-9b9d-87959cfa30d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g4cmx\" (UID: \"30b22991-ec04-4b85-9b9d-87959cfa30d1\") " pod="openshift-multus/multus-additional-cni-plugins-g4cmx"
Sep 17 00:09:48.610775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610544    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs\") pod \"network-metrics-daemon-xbhz5\" (UID: \"b956a62e-c4ac-4e1d-aeb0-3f7907a0a702\") " pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:09:48.610775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610572    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-multus-socket-dir-parent\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.610775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610596    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-etc-modprobe-d\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.610775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610632    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrh5n\" (UniqueName: \"kubernetes.io/projected/30b22991-ec04-4b85-9b9d-87959cfa30d1-kube-api-access-rrh5n\") pod \"multus-additional-cni-plugins-g4cmx\" (UID: \"30b22991-ec04-4b85-9b9d-87959cfa30d1\") " pod="openshift-multus/multus-additional-cni-plugins-g4cmx"
Sep 17 00:09:48.611593 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610673    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grbmg\" (UniqueName: \"kubernetes.io/projected/3e97bfc2-d46a-4384-bc1a-a1a2d37afe75-kube-api-access-grbmg\") pod \"node-resolver-9xxfq\" (UID: \"3e97bfc2-d46a-4384-bc1a-a1a2d37afe75\") " pod="openshift-dns/node-resolver-9xxfq"
Sep 17 00:09:48.611593 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-system-cni-dir\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.611593 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610814    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-cnibin\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.611593 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610843    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-host-run-netns\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.611593 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:48.610857    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf1e8714548f742e80d275d4df6599c.slice/crio-c276bbac1f02ae8763a512337ec135e1a12a96621580025aed5a7d08cde08fa8 WatchSource:0}: Error finding container c276bbac1f02ae8763a512337ec135e1a12a96621580025aed5a7d08cde08fa8: Status 404 returned error can't find the container with id c276bbac1f02ae8763a512337ec135e1a12a96621580025aed5a7d08cde08fa8
Sep 17 00:09:48.611593 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610885    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d261c1cf-76d0-468c-a122-1c2b359f3765-ovnkube-script-lib\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.611593 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-host-var-lib-cni-bin\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.611593 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.610973    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e6570468-fe3a-4512-8bda-9880edc3ff96-serviceca\") pod \"node-ca-rg8zr\" (UID: \"e6570468-fe3a-4512-8bda-9880edc3ff96\") " pod="openshift-image-registry/node-ca-rg8zr"
Sep 17 00:09:48.611593 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.611022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-host-run-multus-certs\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.611593 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.611063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs5sk\" (UniqueName: \"kubernetes.io/projected/f5f1a947-783b-4a35-a70b-b850883fef06-kube-api-access-gs5sk\") pod \"network-check-target-s8lnt\" (UID: \"f5f1a947-783b-4a35-a70b-b850883fef06\") " pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:09:48.611593 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.611093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-etc-sysctl-d\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.611593 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.611178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-etc-systemd\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.611593 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.611211    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-tmp\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.611593 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.611285    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-multus-cni-dir\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.611593 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.611316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-host-var-lib-kubelet\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.611593 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.611365    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-etc-sysconfig\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.612408 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.611391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30b22991-ec04-4b85-9b9d-87959cfa30d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g4cmx\" (UID: \"30b22991-ec04-4b85-9b9d-87959cfa30d1\") " pod="openshift-multus/multus-additional-cni-plugins-g4cmx"
Sep 17 00:09:48.612408 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.611481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a948860-0f14-46ec-afab-e23492108eaa-cni-binary-copy\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.612408 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.611507    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a948860-0f14-46ec-afab-e23492108eaa-multus-daemon-config\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.612408 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.611539    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c956d3aa-d437-4b53-afc6-56127f4e1688-agent-certs\") pod \"konnectivity-agent-w8ps8\" (UID: \"c956d3aa-d437-4b53-afc6-56127f4e1688\") " pod="kube-system/konnectivity-agent-w8ps8"
Sep 17 00:09:48.612408 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.611595    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7mmw\" (UniqueName: \"kubernetes.io/projected/e6570468-fe3a-4512-8bda-9880edc3ff96-kube-api-access-r7mmw\") pod \"node-ca-rg8zr\" (UID: \"e6570468-fe3a-4512-8bda-9880edc3ff96\") " pod="openshift-image-registry/node-ca-rg8zr"
Sep 17 00:09:48.612408 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.611623    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-run-systemd\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.612408 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.611647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-node-log\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.612408 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.611672    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d261c1cf-76d0-468c-a122-1c2b359f3765-ovnkube-config\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.612408 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.611696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjv9q\" (UniqueName: \"kubernetes.io/projected/d261c1cf-76d0-468c-a122-1c2b359f3765-kube-api-access-kjv9q\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.612408 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.611723    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-etc-sysctl-conf\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.612408 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.611750    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3e97bfc2-d46a-4384-bc1a-a1a2d37afe75-hosts-file\") pod \"node-resolver-9xxfq\" (UID: \"3e97bfc2-d46a-4384-bc1a-a1a2d37afe75\") " pod="openshift-dns/node-resolver-9xxfq"
Sep 17 00:09:48.612408 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.611776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-var-lib-openvswitch\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.612408 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.611801    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lph5r\" (UniqueName: \"kubernetes.io/projected/0741efde-4990-4669-9939-d5637d40f0e6-kube-api-access-lph5r\") pod \"aws-ebs-csi-driver-node-g5bpl\" (UID: \"0741efde-4990-4669-9939-d5637d40f0e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl"
Sep 17 00:09:48.612408 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.611825    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-host-var-lib-cni-multus\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.616847 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.616828    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 00:09:48.712338 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712303    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-cnibin\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.712338 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-host-run-netns\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.712594 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712356    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d261c1cf-76d0-468c-a122-1c2b359f3765-ovnkube-script-lib\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.712594 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712373    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-host-var-lib-cni-bin\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.712594 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e6570468-fe3a-4512-8bda-9880edc3ff96-serviceca\") pod \"node-ca-rg8zr\" (UID: \"e6570468-fe3a-4512-8bda-9880edc3ff96\") " pod="openshift-image-registry/node-ca-rg8zr"
Sep 17 00:09:48.712594 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-host-run-multus-certs\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.712594 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712448    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-cnibin\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.712594 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712494    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-host-run-multus-certs\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.712594 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712448    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-host-run-netns\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.712594 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs5sk\" (UniqueName: \"kubernetes.io/projected/f5f1a947-783b-4a35-a70b-b850883fef06-kube-api-access-gs5sk\") pod \"network-check-target-s8lnt\" (UID: \"f5f1a947-783b-4a35-a70b-b850883fef06\") " pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:09:48.712594 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-etc-sysctl-d\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.712594 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-etc-systemd\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.712594 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712569    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-tmp\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.712594 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712584    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-multus-cni-dir\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.713475 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-host-var-lib-kubelet\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.713475 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712623    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-etc-sysconfig\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.713475 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712637    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-host-var-lib-cni-bin\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.713475 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712681    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-etc-sysconfig\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.713475 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712698    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-multus-cni-dir\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.713475 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712707    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-etc-sysctl-d\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.713475 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712704    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-etc-systemd\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.713475 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712725    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-host-var-lib-kubelet\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.713475 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30b22991-ec04-4b85-9b9d-87959cfa30d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g4cmx\" (UID: \"30b22991-ec04-4b85-9b9d-87959cfa30d1\") " pod="openshift-multus/multus-additional-cni-plugins-g4cmx"
Sep 17 00:09:48.713475 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a948860-0f14-46ec-afab-e23492108eaa-cni-binary-copy\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.713475 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712805    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a948860-0f14-46ec-afab-e23492108eaa-multus-daemon-config\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.713475 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.712960    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c956d3aa-d437-4b53-afc6-56127f4e1688-agent-certs\") pod \"konnectivity-agent-w8ps8\" (UID: \"c956d3aa-d437-4b53-afc6-56127f4e1688\") " pod="kube-system/konnectivity-agent-w8ps8"
Sep 17 00:09:48.713475 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713081    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7mmw\" (UniqueName: \"kubernetes.io/projected/e6570468-fe3a-4512-8bda-9880edc3ff96-kube-api-access-r7mmw\") pod \"node-ca-rg8zr\" (UID: \"e6570468-fe3a-4512-8bda-9880edc3ff96\") " pod="openshift-image-registry/node-ca-rg8zr"
Sep 17 00:09:48.713475 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-run-systemd\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.713475 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713420    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 00:09:48.713475 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-node-log\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.713475 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713479    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a948860-0f14-46ec-afab-e23492108eaa-cni-binary-copy\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.714262 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d261c1cf-76d0-468c-a122-1c2b359f3765-ovnkube-config\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.714262 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713199    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e6570468-fe3a-4512-8bda-9880edc3ff96-serviceca\") pod \"node-ca-rg8zr\" (UID: \"e6570468-fe3a-4512-8bda-9880edc3ff96\") " pod="openshift-image-registry/node-ca-rg8zr"
Sep 17 00:09:48.714262 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713137    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d261c1cf-76d0-468c-a122-1c2b359f3765-ovnkube-script-lib\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.714262 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713540    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjv9q\" (UniqueName: \"kubernetes.io/projected/d261c1cf-76d0-468c-a122-1c2b359f3765-kube-api-access-kjv9q\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.714262 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713561    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-run-systemd\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.714262 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713601    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-node-log\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.714262 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713615    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30b22991-ec04-4b85-9b9d-87959cfa30d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g4cmx\" (UID: \"30b22991-ec04-4b85-9b9d-87959cfa30d1\") " pod="openshift-multus/multus-additional-cni-plugins-g4cmx"
Sep 17 00:09:48.714262 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713565    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-etc-sysctl-conf\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.714262 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713667    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3e97bfc2-d46a-4384-bc1a-a1a2d37afe75-hosts-file\") pod \"node-resolver-9xxfq\" (UID: \"3e97bfc2-d46a-4384-bc1a-a1a2d37afe75\") " pod="openshift-dns/node-resolver-9xxfq"
Sep 17 00:09:48.714262 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713692    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-etc-sysctl-conf\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.714262 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713692    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-var-lib-openvswitch\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.714262 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713724    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-var-lib-openvswitch\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.714262 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713739    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lph5r\" (UniqueName: \"kubernetes.io/projected/0741efde-4990-4669-9939-d5637d40f0e6-kube-api-access-lph5r\") pod \"aws-ebs-csi-driver-node-g5bpl\" (UID: \"0741efde-4990-4669-9939-d5637d40f0e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl"
Sep 17 00:09:48.714262 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-host-var-lib-cni-multus\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.714262 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713775    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3e97bfc2-d46a-4384-bc1a-a1a2d37afe75-hosts-file\") pod \"node-resolver-9xxfq\" (UID: \"3e97bfc2-d46a-4384-bc1a-a1a2d37afe75\") " pod="openshift-dns/node-resolver-9xxfq"
Sep 17 00:09:48.714262 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-sys\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.714262 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713822    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-lib-modules\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.715024 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713828    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-host-var-lib-cni-multus\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.715024 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-etc-tuned\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.715024 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713858    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-sys\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.715024 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713878    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0741efde-4990-4669-9939-d5637d40f0e6-device-dir\") pod \"aws-ebs-csi-driver-node-g5bpl\" (UID: \"0741efde-4990-4669-9939-d5637d40f0e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl"
Sep 17 00:09:48.715024 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0741efde-4990-4669-9939-d5637d40f0e6-etc-selinux\") pod \"aws-ebs-csi-driver-node-g5bpl\" (UID: \"0741efde-4990-4669-9939-d5637d40f0e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl"
Sep 17 00:09:48.715024 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713924    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/504ae0de-dca4-48aa-964c-b457c02267a5-original-pull-secret\") pod \"global-pull-secret-syncer-ch5w2\" (UID: \"504ae0de-dca4-48aa-964c-b457c02267a5\") " pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:09:48.715024 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713946    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-systemd-units\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.715024 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-host-run-ovn-kubernetes\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.715024 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0741efde-4990-4669-9939-d5637d40f0e6-sys-fs\") pod \"aws-ebs-csi-driver-node-g5bpl\" (UID: \"0741efde-4990-4669-9939-d5637d40f0e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl"
Sep 17 00:09:48.715024 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713983    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d261c1cf-76d0-468c-a122-1c2b359f3765-ovnkube-config\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.715024 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.713998    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-lib-modules\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.715024 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-host-run-netns\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.715024 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714041    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-host-run-netns\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.715024 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714040    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-multus-conf-dir\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.715024 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxnfj\" (UniqueName: \"kubernetes.io/projected/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-kube-api-access-bxnfj\") pod \"network-metrics-daemon-xbhz5\" (UID: \"b956a62e-c4ac-4e1d-aeb0-3f7907a0a702\") " pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:09:48.715024 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714073    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-multus-conf-dir\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.715024 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714103    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a948860-0f14-46ec-afab-e23492108eaa-multus-daemon-config\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.715024 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:48.714113    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 00:09:48.716117 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714108    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d261c1cf-76d0-468c-a122-1c2b359f3765-env-overrides\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.716117 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714155    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0741efde-4990-4669-9939-d5637d40f0e6-sys-fs\") pod \"aws-ebs-csi-driver-node-g5bpl\" (UID: \"0741efde-4990-4669-9939-d5637d40f0e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl"
Sep 17 00:09:48.716117 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-var-lib-kubelet\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.716117 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:48.714177    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/504ae0de-dca4-48aa-964c-b457c02267a5-original-pull-secret podName:504ae0de-dca4-48aa-964c-b457c02267a5 nodeName:}" failed. No retries permitted until 2026-09-17 00:09:49.214156153 +0000 UTC m=+2.072649283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/504ae0de-dca4-48aa-964c-b457c02267a5-original-pull-secret") pod "global-pull-secret-syncer-ch5w2" (UID: "504ae0de-dca4-48aa-964c-b457c02267a5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 00:09:48.716117 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714202    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-var-lib-kubelet\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.716117 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714207    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkpq2\" (UniqueName: \"kubernetes.io/projected/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-kube-api-access-rkpq2\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.716117 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714235    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0741efde-4990-4669-9939-d5637d40f0e6-etc-selinux\") pod \"aws-ebs-csi-driver-node-g5bpl\" (UID: \"0741efde-4990-4669-9939-d5637d40f0e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl"
Sep 17 00:09:48.716117 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5685bd44-e964-4176-ae6e-479bcee01cbb-iptables-alerter-script\") pod \"iptables-alerter-4bds9\" (UID: \"5685bd44-e964-4176-ae6e-479bcee01cbb\") " pod="openshift-network-operator/iptables-alerter-4bds9"
Sep 17 00:09:48.716117 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714250    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-systemd-units\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.716117 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714118    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-host-run-ovn-kubernetes\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.716117 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714291    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0741efde-4990-4669-9939-d5637d40f0e6-device-dir\") pod \"aws-ebs-csi-driver-node-g5bpl\" (UID: \"0741efde-4990-4669-9939-d5637d40f0e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl"
Sep 17 00:09:48.716117 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4v9p\" (UniqueName: \"kubernetes.io/projected/5685bd44-e964-4176-ae6e-479bcee01cbb-kube-api-access-n4v9p\") pod \"iptables-alerter-4bds9\" (UID: \"5685bd44-e964-4176-ae6e-479bcee01cbb\") " pod="openshift-network-operator/iptables-alerter-4bds9"
Sep 17 00:09:48.716117 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/504ae0de-dca4-48aa-964c-b457c02267a5-kubelet-config\") pod \"global-pull-secret-syncer-ch5w2\" (UID: \"504ae0de-dca4-48aa-964c-b457c02267a5\") " pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:09:48.716117 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-run-ovn\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.716117 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714522    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0741efde-4990-4669-9939-d5637d40f0e6-socket-dir\") pod \"aws-ebs-csi-driver-node-g5bpl\" (UID: \"0741efde-4990-4669-9939-d5637d40f0e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl"
Sep 17 00:09:48.716117 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhwz4\" (UniqueName: \"kubernetes.io/projected/0a948860-0f14-46ec-afab-e23492108eaa-kube-api-access-zhwz4\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.716988 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714547    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/504ae0de-dca4-48aa-964c-b457c02267a5-kubelet-config\") pod \"global-pull-secret-syncer-ch5w2\" (UID: \"504ae0de-dca4-48aa-964c-b457c02267a5\") " pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:09:48.716988 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714587    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c956d3aa-d437-4b53-afc6-56127f4e1688-konnectivity-ca\") pod \"konnectivity-agent-w8ps8\" (UID: \"c956d3aa-d437-4b53-afc6-56127f4e1688\") " pod="kube-system/konnectivity-agent-w8ps8"
Sep 17 00:09:48.716988 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714611    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-host-slash\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.716988 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-host-cni-netd\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.716988 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714588    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-run-ovn\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.716988 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714690    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d261c1cf-76d0-468c-a122-1c2b359f3765-env-overrides\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.716988 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.716988 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d261c1cf-76d0-468c-a122-1c2b359f3765-ovn-node-metrics-cert\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.716988 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714761    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-etc-kubernetes\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.716988 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-host\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.716988 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714853    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-host\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.716988 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714897    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-etc-kubernetes\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.716988 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714934    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-host-slash\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.716988 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.714969    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.716988 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715060    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-host-cni-netd\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.716988 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715085    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5685bd44-e964-4176-ae6e-479bcee01cbb-iptables-alerter-script\") pod \"iptables-alerter-4bds9\" (UID: \"5685bd44-e964-4176-ae6e-479bcee01cbb\") " pod="openshift-network-operator/iptables-alerter-4bds9"
Sep 17 00:09:48.716988 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715177    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0741efde-4990-4669-9939-d5637d40f0e6-socket-dir\") pod \"aws-ebs-csi-driver-node-g5bpl\" (UID: \"0741efde-4990-4669-9939-d5637d40f0e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl"
Sep 17 00:09:48.717803 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715266    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30b22991-ec04-4b85-9b9d-87959cfa30d1-system-cni-dir\") pod \"multus-additional-cni-plugins-g4cmx\" (UID: \"30b22991-ec04-4b85-9b9d-87959cfa30d1\") " pod="openshift-multus/multus-additional-cni-plugins-g4cmx"
Sep 17 00:09:48.717803 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715287    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c956d3aa-d437-4b53-afc6-56127f4e1688-konnectivity-ca\") pod \"konnectivity-agent-w8ps8\" (UID: \"c956d3aa-d437-4b53-afc6-56127f4e1688\") " pod="kube-system/konnectivity-agent-w8ps8"
Sep 17 00:09:48.717803 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715303    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0741efde-4990-4669-9939-d5637d40f0e6-registration-dir\") pod \"aws-ebs-csi-driver-node-g5bpl\" (UID: \"0741efde-4990-4669-9939-d5637d40f0e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl"
Sep 17 00:09:48.717803 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-host-kubelet\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.717803 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715364    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/504ae0de-dca4-48aa-964c-b457c02267a5-dbus\") pod \"global-pull-secret-syncer-ch5w2\" (UID: \"504ae0de-dca4-48aa-964c-b457c02267a5\") " pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:09:48.717803 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715367    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30b22991-ec04-4b85-9b9d-87959cfa30d1-system-cni-dir\") pod \"multus-additional-cni-plugins-g4cmx\" (UID: \"30b22991-ec04-4b85-9b9d-87959cfa30d1\") " pod="openshift-multus/multus-additional-cni-plugins-g4cmx"
Sep 17 00:09:48.717803 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-run-openvswitch\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.717803 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-host-cni-bin\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.717803 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715412    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-host-kubelet\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.717803 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30b22991-ec04-4b85-9b9d-87959cfa30d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-g4cmx\" (UID: \"30b22991-ec04-4b85-9b9d-87959cfa30d1\") " pod="openshift-multus/multus-additional-cni-plugins-g4cmx"
Sep 17 00:09:48.717803 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715471    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30b22991-ec04-4b85-9b9d-87959cfa30d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g4cmx\" (UID: \"30b22991-ec04-4b85-9b9d-87959cfa30d1\") " pod="openshift-multus/multus-additional-cni-plugins-g4cmx"
Sep 17 00:09:48.717803 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715481    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0741efde-4990-4669-9939-d5637d40f0e6-registration-dir\") pod \"aws-ebs-csi-driver-node-g5bpl\" (UID: \"0741efde-4990-4669-9939-d5637d40f0e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl"
Sep 17 00:09:48.717803 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715512    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0741efde-4990-4669-9939-d5637d40f0e6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g5bpl\" (UID: \"0741efde-4990-4669-9939-d5637d40f0e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl"
Sep 17 00:09:48.717803 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715559    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-host-cni-bin\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.717803 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715578    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0741efde-4990-4669-9939-d5637d40f0e6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g5bpl\" (UID: \"0741efde-4990-4669-9939-d5637d40f0e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl"
Sep 17 00:09:48.717803 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715591    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-run\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.717803 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715618    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30b22991-ec04-4b85-9b9d-87959cfa30d1-cnibin\") pod \"multus-additional-cni-plugins-g4cmx\" (UID: \"30b22991-ec04-4b85-9b9d-87959cfa30d1\") " pod="openshift-multus/multus-additional-cni-plugins-g4cmx"
Sep 17 00:09:48.718327 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e97bfc2-d46a-4384-bc1a-a1a2d37afe75-tmp-dir\") pod \"node-resolver-9xxfq\" (UID: \"3e97bfc2-d46a-4384-bc1a-a1a2d37afe75\") " pod="openshift-dns/node-resolver-9xxfq"
Sep 17 00:09:48.718327 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-host-run-k8s-cni-cncf-io\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.718327 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715721    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-hostroot\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.718327 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-etc-kubernetes\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.718327 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6570468-fe3a-4512-8bda-9880edc3ff96-host\") pod \"node-ca-rg8zr\" (UID: \"e6570468-fe3a-4512-8bda-9880edc3ff96\") " pod="openshift-image-registry/node-ca-rg8zr"
Sep 17 00:09:48.718327 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715812    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5685bd44-e964-4176-ae6e-479bcee01cbb-host-slash\") pod \"iptables-alerter-4bds9\" (UID: \"5685bd44-e964-4176-ae6e-479bcee01cbb\") " pod="openshift-network-operator/iptables-alerter-4bds9"
Sep 17 00:09:48.718327 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-etc-openvswitch\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.718327 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-log-socket\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.718327 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30b22991-ec04-4b85-9b9d-87959cfa30d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g4cmx\" (UID: \"30b22991-ec04-4b85-9b9d-87959cfa30d1\") " pod="openshift-multus/multus-additional-cni-plugins-g4cmx"
Sep 17 00:09:48.718327 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs\") pod \"network-metrics-daemon-xbhz5\" (UID: \"b956a62e-c4ac-4e1d-aeb0-3f7907a0a702\") " pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:09:48.718327 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715943    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-multus-socket-dir-parent\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.718327 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.715970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-etc-modprobe-d\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.718327 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.716022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrh5n\" (UniqueName: \"kubernetes.io/projected/30b22991-ec04-4b85-9b9d-87959cfa30d1-kube-api-access-rrh5n\") pod \"multus-additional-cni-plugins-g4cmx\" (UID: \"30b22991-ec04-4b85-9b9d-87959cfa30d1\") " pod="openshift-multus/multus-additional-cni-plugins-g4cmx"
Sep 17 00:09:48.718327 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.716036    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30b22991-ec04-4b85-9b9d-87959cfa30d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-g4cmx\" (UID: \"30b22991-ec04-4b85-9b9d-87959cfa30d1\") " pod="openshift-multus/multus-additional-cni-plugins-g4cmx"
Sep 17 00:09:48.718327 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.716051    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grbmg\" (UniqueName: \"kubernetes.io/projected/3e97bfc2-d46a-4384-bc1a-a1a2d37afe75-kube-api-access-grbmg\") pod \"node-resolver-9xxfq\" (UID: \"3e97bfc2-d46a-4384-bc1a-a1a2d37afe75\") " pod="openshift-dns/node-resolver-9xxfq"
Sep 17 00:09:48.718327 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.716082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-system-cni-dir\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.718327 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.716165    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-system-cni-dir\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.718327 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.716295    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/504ae0de-dca4-48aa-964c-b457c02267a5-dbus\") pod \"global-pull-secret-syncer-ch5w2\" (UID: \"504ae0de-dca4-48aa-964c-b457c02267a5\") " pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:09:48.718910 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.716342    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-run-openvswitch\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.718910 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.716391    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-run\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.718910 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.716414    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5685bd44-e964-4176-ae6e-479bcee01cbb-host-slash\") pod \"iptables-alerter-4bds9\" (UID: \"5685bd44-e964-4176-ae6e-479bcee01cbb\") " pod="openshift-network-operator/iptables-alerter-4bds9"
Sep 17 00:09:48.718910 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.716434    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-etc-openvswitch\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.718910 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.716484    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d261c1cf-76d0-468c-a122-1c2b359f3765-log-socket\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.718910 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.716668    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30b22991-ec04-4b85-9b9d-87959cfa30d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g4cmx\" (UID: \"30b22991-ec04-4b85-9b9d-87959cfa30d1\") " pod="openshift-multus/multus-additional-cni-plugins-g4cmx"
Sep 17 00:09:48.718910 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.716671    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30b22991-ec04-4b85-9b9d-87959cfa30d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g4cmx\" (UID: \"30b22991-ec04-4b85-9b9d-87959cfa30d1\") " pod="openshift-multus/multus-additional-cni-plugins-g4cmx"
Sep 17 00:09:48.718910 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.716724    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-multus-socket-dir-parent\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.718910 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.716740    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e97bfc2-d46a-4384-bc1a-a1a2d37afe75-tmp-dir\") pod \"node-resolver-9xxfq\" (UID: \"3e97bfc2-d46a-4384-bc1a-a1a2d37afe75\") " pod="openshift-dns/node-resolver-9xxfq"
Sep 17 00:09:48.718910 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.716772    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-etc-modprobe-d\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.718910 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:48.716804    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:09:48.718910 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.716819    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30b22991-ec04-4b85-9b9d-87959cfa30d1-cnibin\") pod \"multus-additional-cni-plugins-g4cmx\" (UID: \"30b22991-ec04-4b85-9b9d-87959cfa30d1\") " pod="openshift-multus/multus-additional-cni-plugins-g4cmx"
Sep 17 00:09:48.718910 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:48.716853    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs podName:b956a62e-c4ac-4e1d-aeb0-3f7907a0a702 nodeName:}" failed. No retries permitted until 2026-09-17 00:09:49.216836822 +0000 UTC m=+2.075329955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs") pod "network-metrics-daemon-xbhz5" (UID: "b956a62e-c4ac-4e1d-aeb0-3f7907a0a702") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:09:48.718910 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.716867    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-host-run-k8s-cni-cncf-io\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.718910 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.716899    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-etc-kubernetes\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.718910 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.716910    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6570468-fe3a-4512-8bda-9880edc3ff96-host\") pod \"node-ca-rg8zr\" (UID: \"e6570468-fe3a-4512-8bda-9880edc3ff96\") " pod="openshift-image-registry/node-ca-rg8zr"
Sep 17 00:09:48.718910 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.716939    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a948860-0f14-46ec-afab-e23492108eaa-hostroot\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.718910 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.717740    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-tmp\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.719469 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.717800    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-etc-tuned\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.719469 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.717941    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d261c1cf-76d0-468c-a122-1c2b359f3765-ovn-node-metrics-cert\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.719469 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.718070    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c956d3aa-d437-4b53-afc6-56127f4e1688-agent-certs\") pod \"konnectivity-agent-w8ps8\" (UID: \"c956d3aa-d437-4b53-afc6-56127f4e1688\") " pod="kube-system/konnectivity-agent-w8ps8"
Sep 17 00:09:48.719469 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:48.719109    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 00:09:48.719469 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:48.719129    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 00:09:48.719469 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:48.719146    2577 projected.go:196] Error preparing data for projected volume kube-api-access-gs5sk for pod openshift-network-diagnostics/network-check-target-s8lnt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:09:48.719469 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:48.719196    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5f1a947-783b-4a35-a70b-b850883fef06-kube-api-access-gs5sk podName:f5f1a947-783b-4a35-a70b-b850883fef06 nodeName:}" failed. No retries permitted until 2026-09-17 00:09:49.219181419 +0000 UTC m=+2.077674541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gs5sk" (UniqueName: "kubernetes.io/projected/f5f1a947-783b-4a35-a70b-b850883fef06-kube-api-access-gs5sk") pod "network-check-target-s8lnt" (UID: "f5f1a947-783b-4a35-a70b-b850883fef06") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:09:48.723476 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.723395    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lph5r\" (UniqueName: \"kubernetes.io/projected/0741efde-4990-4669-9939-d5637d40f0e6-kube-api-access-lph5r\") pod \"aws-ebs-csi-driver-node-g5bpl\" (UID: \"0741efde-4990-4669-9939-d5637d40f0e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl"
Sep 17 00:09:48.724415 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.724390    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjv9q\" (UniqueName: \"kubernetes.io/projected/d261c1cf-76d0-468c-a122-1c2b359f3765-kube-api-access-kjv9q\") pod \"ovnkube-node-bz685\" (UID: \"d261c1cf-76d0-468c-a122-1c2b359f3765\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.725038 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.725004    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxnfj\" (UniqueName: \"kubernetes.io/projected/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-kube-api-access-bxnfj\") pod \"network-metrics-daemon-xbhz5\" (UID: \"b956a62e-c4ac-4e1d-aeb0-3f7907a0a702\") " pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:09:48.725139 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.725106    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7mmw\" (UniqueName: \"kubernetes.io/projected/e6570468-fe3a-4512-8bda-9880edc3ff96-kube-api-access-r7mmw\") pod \"node-ca-rg8zr\" (UID: \"e6570468-fe3a-4512-8bda-9880edc3ff96\") " pod="openshift-image-registry/node-ca-rg8zr"
Sep 17 00:09:48.725712 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.725686    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrh5n\" (UniqueName: \"kubernetes.io/projected/30b22991-ec04-4b85-9b9d-87959cfa30d1-kube-api-access-rrh5n\") pod \"multus-additional-cni-plugins-g4cmx\" (UID: \"30b22991-ec04-4b85-9b9d-87959cfa30d1\") " pod="openshift-multus/multus-additional-cni-plugins-g4cmx"
Sep 17 00:09:48.725712 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.725703    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkpq2\" (UniqueName: \"kubernetes.io/projected/924b0fd5-7d3d-49cd-9ecb-bd4430ecec27-kube-api-access-rkpq2\") pod \"tuned-vml9z\" (UID: \"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27\") " pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.725855 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.725786    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4v9p\" (UniqueName: \"kubernetes.io/projected/5685bd44-e964-4176-ae6e-479bcee01cbb-kube-api-access-n4v9p\") pod \"iptables-alerter-4bds9\" (UID: \"5685bd44-e964-4176-ae6e-479bcee01cbb\") " pod="openshift-network-operator/iptables-alerter-4bds9"
Sep 17 00:09:48.726157 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.726139    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grbmg\" (UniqueName: \"kubernetes.io/projected/3e97bfc2-d46a-4384-bc1a-a1a2d37afe75-kube-api-access-grbmg\") pod \"node-resolver-9xxfq\" (UID: \"3e97bfc2-d46a-4384-bc1a-a1a2d37afe75\") " pod="openshift-dns/node-resolver-9xxfq"
Sep 17 00:09:48.727244 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.727227    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhwz4\" (UniqueName: \"kubernetes.io/projected/0a948860-0f14-46ec-afab-e23492108eaa-kube-api-access-zhwz4\") pod \"multus-k2t7f\" (UID: \"0a948860-0f14-46ec-afab-e23492108eaa\") " pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.816077 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.815995    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 00:09:48.833241 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.833213    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-w8ps8"
Sep 17 00:09:48.843434 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:48.843255    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc956d3aa_d437_4b53_afc6_56127f4e1688.slice/crio-b64e0fcc23745f50800438ef3539a8dcbaccbf3370a675800bc1d538248373d7 WatchSource:0}: Error finding container b64e0fcc23745f50800438ef3539a8dcbaccbf3370a675800bc1d538248373d7: Status 404 returned error can't find the container with id b64e0fcc23745f50800438ef3539a8dcbaccbf3370a675800bc1d538248373d7
Sep 17 00:09:48.851895 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.851871    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4bds9"
Sep 17 00:09:48.864850 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.864831    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:09:48.869823 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.869802    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vml9z"
Sep 17 00:09:48.872832 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:48.872806    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd261c1cf_76d0_468c_a122_1c2b359f3765.slice/crio-e385a07006df0491ef724ca7d1c4cff0d9b986e6d772d29fd1a2a7b0cf114851 WatchSource:0}: Error finding container e385a07006df0491ef724ca7d1c4cff0d9b986e6d772d29fd1a2a7b0cf114851: Status 404 returned error can't find the container with id e385a07006df0491ef724ca7d1c4cff0d9b986e6d772d29fd1a2a7b0cf114851
Sep 17 00:09:48.877088 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:48.877067    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924b0fd5_7d3d_49cd_9ecb_bd4430ecec27.slice/crio-0dd840c3641b79f91b9e0a6a74e08e37749241b538ff26dbd1d524a92c0043be WatchSource:0}: Error finding container 0dd840c3641b79f91b9e0a6a74e08e37749241b538ff26dbd1d524a92c0043be: Status 404 returned error can't find the container with id 0dd840c3641b79f91b9e0a6a74e08e37749241b538ff26dbd1d524a92c0043be
Sep 17 00:09:48.891638 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.891616    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rg8zr"
Sep 17 00:09:48.899229 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:48.899201    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6570468_fe3a_4512_8bda_9880edc3ff96.slice/crio-8d4e5e7cc65abd2120552220a0aad02089bb0f78eb744448a4af33ee594966bb WatchSource:0}: Error finding container 8d4e5e7cc65abd2120552220a0aad02089bb0f78eb744448a4af33ee594966bb: Status 404 returned error can't find the container with id 8d4e5e7cc65abd2120552220a0aad02089bb0f78eb744448a4af33ee594966bb
Sep 17 00:09:48.909923 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.909903    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g4cmx"
Sep 17 00:09:48.917124 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:48.917103    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b22991_ec04_4b85_9b9d_87959cfa30d1.slice/crio-96e31a6ee8b7c2a0c867796770f8cc928d05e8e6d68ce5888fb5b7a9982d1893 WatchSource:0}: Error finding container 96e31a6ee8b7c2a0c867796770f8cc928d05e8e6d68ce5888fb5b7a9982d1893: Status 404 returned error can't find the container with id 96e31a6ee8b7c2a0c867796770f8cc928d05e8e6d68ce5888fb5b7a9982d1893
Sep 17 00:09:48.917622 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.917606    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9xxfq"
Sep 17 00:09:48.924952 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.924931    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl"
Sep 17 00:09:48.925134 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:48.925111    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e97bfc2_d46a_4384_bc1a_a1a2d37afe75.slice/crio-6b736151c9fdb5d387ae37f219a063f2a64425a5662ca929f220e44c410fec22 WatchSource:0}: Error finding container 6b736151c9fdb5d387ae37f219a063f2a64425a5662ca929f220e44c410fec22: Status 404 returned error can't find the container with id 6b736151c9fdb5d387ae37f219a063f2a64425a5662ca929f220e44c410fec22
Sep 17 00:09:48.932207 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:48.932180    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k2t7f"
Sep 17 00:09:48.933988 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:48.933964    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0741efde_4990_4669_9939_d5637d40f0e6.slice/crio-1ba25b3fc0a9d3403b4e9be8bb58c08cf734ac505afb2a01dd69ac1af26083fe WatchSource:0}: Error finding container 1ba25b3fc0a9d3403b4e9be8bb58c08cf734ac505afb2a01dd69ac1af26083fe: Status 404 returned error can't find the container with id 1ba25b3fc0a9d3403b4e9be8bb58c08cf734ac505afb2a01dd69ac1af26083fe
Sep 17 00:09:48.939683 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:09:48.939663    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a948860_0f14_46ec_afab_e23492108eaa.slice/crio-3ebe2fda532f7d23b796ff6241e0b6d0657b1c5c90c49a4f0522c5aac0c19361 WatchSource:0}: Error finding container 3ebe2fda532f7d23b796ff6241e0b6d0657b1c5c90c49a4f0522c5aac0c19361: Status 404 returned error can't find the container with id 3ebe2fda532f7d23b796ff6241e0b6d0657b1c5c90c49a4f0522c5aac0c19361
Sep 17 00:09:49.099497 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:49.099407    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 00:09:49.219601 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:49.219569    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/504ae0de-dca4-48aa-964c-b457c02267a5-original-pull-secret\") pod \"global-pull-secret-syncer-ch5w2\" (UID: \"504ae0de-dca4-48aa-964c-b457c02267a5\") " pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:09:49.219792 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:49.219638    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs\") pod \"network-metrics-daemon-xbhz5\" (UID: \"b956a62e-c4ac-4e1d-aeb0-3f7907a0a702\") " pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:09:49.219792 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:49.219671    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs5sk\" (UniqueName: \"kubernetes.io/projected/f5f1a947-783b-4a35-a70b-b850883fef06-kube-api-access-gs5sk\") pod \"network-check-target-s8lnt\" (UID: \"f5f1a947-783b-4a35-a70b-b850883fef06\") " pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:09:49.219792 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:49.219769    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 00:09:49.219792 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:49.219784    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:09:49.220005 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:49.219841    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs podName:b956a62e-c4ac-4e1d-aeb0-3f7907a0a702 nodeName:}" failed. No retries permitted until 2026-09-17 00:09:50.219821299 +0000 UTC m=+3.078314430 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs") pod "network-metrics-daemon-xbhz5" (UID: "b956a62e-c4ac-4e1d-aeb0-3f7907a0a702") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:09:49.220005 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:49.219858    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/504ae0de-dca4-48aa-964c-b457c02267a5-original-pull-secret podName:504ae0de-dca4-48aa-964c-b457c02267a5 nodeName:}" failed. No retries permitted until 2026-09-17 00:09:50.219850335 +0000 UTC m=+3.078343449 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/504ae0de-dca4-48aa-964c-b457c02267a5-original-pull-secret") pod "global-pull-secret-syncer-ch5w2" (UID: "504ae0de-dca4-48aa-964c-b457c02267a5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 00:09:49.220005 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:49.219899    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 00:09:49.220005 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:49.219919    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 00:09:49.220005 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:49.219933    2577 projected.go:196] Error preparing data for projected volume kube-api-access-gs5sk for pod openshift-network-diagnostics/network-check-target-s8lnt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:09:49.220005 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:49.219990    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5f1a947-783b-4a35-a70b-b850883fef06-kube-api-access-gs5sk podName:f5f1a947-783b-4a35-a70b-b850883fef06 nodeName:}" failed. No retries permitted until 2026-09-17 00:09:50.219963094 +0000 UTC m=+3.078456209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gs5sk" (UniqueName: "kubernetes.io/projected/f5f1a947-783b-4a35-a70b-b850883fef06-kube-api-access-gs5sk") pod "network-check-target-s8lnt" (UID: "f5f1a947-783b-4a35-a70b-b850883fef06") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:09:49.437911 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:49.437835    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 00:09:49.551790 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:49.551704    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 00:04:49 +0000 UTC" deadline="2028-05-08 15:27:26.396443484 +0000 UTC"
Sep 17 00:09:49.552275 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:49.551798    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14391h17m36.844651725s"
Sep 17 00:09:49.634169 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:49.634064    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9xxfq" event={"ID":"3e97bfc2-d46a-4384-bc1a-a1a2d37afe75","Type":"ContainerStarted","Data":"6b736151c9fdb5d387ae37f219a063f2a64425a5662ca929f220e44c410fec22"}
Sep 17 00:09:49.652728 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:49.652637    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4cmx" event={"ID":"30b22991-ec04-4b85-9b9d-87959cfa30d1","Type":"ContainerStarted","Data":"96e31a6ee8b7c2a0c867796770f8cc928d05e8e6d68ce5888fb5b7a9982d1893"}
Sep 17 00:09:49.663877 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:49.663792    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vml9z" event={"ID":"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27","Type":"ContainerStarted","Data":"0dd840c3641b79f91b9e0a6a74e08e37749241b538ff26dbd1d524a92c0043be"}
Sep 17 00:09:49.675999 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:49.675972    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-w8ps8" event={"ID":"c956d3aa-d437-4b53-afc6-56127f4e1688","Type":"ContainerStarted","Data":"b64e0fcc23745f50800438ef3539a8dcbaccbf3370a675800bc1d538248373d7"}
Sep 17 00:09:49.685752 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:49.685666    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-27.ec2.internal" event={"ID":"fbf1e8714548f742e80d275d4df6599c","Type":"ContainerStarted","Data":"c276bbac1f02ae8763a512337ec135e1a12a96621580025aed5a7d08cde08fa8"}
Sep 17 00:09:49.698753 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:49.698682    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-27.ec2.internal" event={"ID":"ff3d9d4f106049a4c07b46e4d77c855d","Type":"ContainerStarted","Data":"34f75828f2cb62cf5de81863cccdc659cea38cfaee6b9f60351ae512bd718556"}
Sep 17 00:09:49.701640 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:49.701610    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k2t7f" event={"ID":"0a948860-0f14-46ec-afab-e23492108eaa","Type":"ContainerStarted","Data":"3ebe2fda532f7d23b796ff6241e0b6d0657b1c5c90c49a4f0522c5aac0c19361"}
Sep 17 00:09:49.715336 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:49.715288    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl" event={"ID":"0741efde-4990-4669-9939-d5637d40f0e6","Type":"ContainerStarted","Data":"1ba25b3fc0a9d3403b4e9be8bb58c08cf734ac505afb2a01dd69ac1af26083fe"}
Sep 17 00:09:49.727732 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:49.727700    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rg8zr" event={"ID":"e6570468-fe3a-4512-8bda-9880edc3ff96","Type":"ContainerStarted","Data":"8d4e5e7cc65abd2120552220a0aad02089bb0f78eb744448a4af33ee594966bb"}
Sep 17 00:09:49.740573 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:49.740540    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bz685" event={"ID":"d261c1cf-76d0-468c-a122-1c2b359f3765","Type":"ContainerStarted","Data":"e385a07006df0491ef724ca7d1c4cff0d9b986e6d772d29fd1a2a7b0cf114851"}
Sep 17 00:09:49.760348 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:49.760302    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4bds9" event={"ID":"5685bd44-e964-4176-ae6e-479bcee01cbb","Type":"ContainerStarted","Data":"6a475db294798d85ce662a3affcd33591d0e0e7d0a5fa74ad763093520ecbddc"}
Sep 17 00:09:50.229608 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:50.229566    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs\") pod \"network-metrics-daemon-xbhz5\" (UID: \"b956a62e-c4ac-4e1d-aeb0-3f7907a0a702\") " pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:09:50.229841 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:50.229627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs5sk\" (UniqueName: \"kubernetes.io/projected/f5f1a947-783b-4a35-a70b-b850883fef06-kube-api-access-gs5sk\") pod \"network-check-target-s8lnt\" (UID: \"f5f1a947-783b-4a35-a70b-b850883fef06\") " pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:09:50.229841 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:50.229672    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/504ae0de-dca4-48aa-964c-b457c02267a5-original-pull-secret\") pod \"global-pull-secret-syncer-ch5w2\" (UID: \"504ae0de-dca4-48aa-964c-b457c02267a5\") " pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:09:50.229841 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:50.229811    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 00:09:50.230018 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:50.229898    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/504ae0de-dca4-48aa-964c-b457c02267a5-original-pull-secret podName:504ae0de-dca4-48aa-964c-b457c02267a5 nodeName:}" failed. No retries permitted until 2026-09-17 00:09:52.229879662 +0000 UTC m=+5.088372796 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/504ae0de-dca4-48aa-964c-b457c02267a5-original-pull-secret") pod "global-pull-secret-syncer-ch5w2" (UID: "504ae0de-dca4-48aa-964c-b457c02267a5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 00:09:50.230594 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:50.230358    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:09:50.230594 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:50.230416    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs podName:b956a62e-c4ac-4e1d-aeb0-3f7907a0a702 nodeName:}" failed. No retries permitted until 2026-09-17 00:09:52.230399784 +0000 UTC m=+5.088892903 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs") pod "network-metrics-daemon-xbhz5" (UID: "b956a62e-c4ac-4e1d-aeb0-3f7907a0a702") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:09:50.230594 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:50.230522    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 00:09:50.230594 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:50.230540    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 00:09:50.230839 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:50.230613    2577 projected.go:196] Error preparing data for projected volume kube-api-access-gs5sk for pod openshift-network-diagnostics/network-check-target-s8lnt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:09:50.230839 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:50.230664    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5f1a947-783b-4a35-a70b-b850883fef06-kube-api-access-gs5sk podName:f5f1a947-783b-4a35-a70b-b850883fef06 nodeName:}" failed. No retries permitted until 2026-09-17 00:09:52.230649231 +0000 UTC m=+5.089142361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gs5sk" (UniqueName: "kubernetes.io/projected/f5f1a947-783b-4a35-a70b-b850883fef06-kube-api-access-gs5sk") pod "network-check-target-s8lnt" (UID: "f5f1a947-783b-4a35-a70b-b850883fef06") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:09:50.552706 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:50.552609    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 00:04:49 +0000 UTC" deadline="2028-05-01 16:48:21.017059831 +0000 UTC"
Sep 17 00:09:50.552706 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:50.552653    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14224h38m30.464412414s"
Sep 17 00:09:50.577513 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:50.577456    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 00:09:50.601802 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:50.601769    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:09:50.601964 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:50.601910    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch5w2" podUID="504ae0de-dca4-48aa-964c-b457c02267a5"
Sep 17 00:09:50.602342 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:50.602322    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:09:50.602463 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:50.602443    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbhz5" podUID="b956a62e-c4ac-4e1d-aeb0-3f7907a0a702"
Sep 17 00:09:50.602555 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:50.602531    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:09:50.602626 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:50.602606    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8lnt" podUID="f5f1a947-783b-4a35-a70b-b850883fef06"
Sep 17 00:09:52.246636 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:52.246536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs\") pod \"network-metrics-daemon-xbhz5\" (UID: \"b956a62e-c4ac-4e1d-aeb0-3f7907a0a702\") " pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:09:52.246636 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:52.246605    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs5sk\" (UniqueName: \"kubernetes.io/projected/f5f1a947-783b-4a35-a70b-b850883fef06-kube-api-access-gs5sk\") pod \"network-check-target-s8lnt\" (UID: \"f5f1a947-783b-4a35-a70b-b850883fef06\") " pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:09:52.247298 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:52.246648    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/504ae0de-dca4-48aa-964c-b457c02267a5-original-pull-secret\") pod \"global-pull-secret-syncer-ch5w2\" (UID: \"504ae0de-dca4-48aa-964c-b457c02267a5\") " pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:09:52.247298 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:52.246782    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 00:09:52.247298 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:52.246846    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/504ae0de-dca4-48aa-964c-b457c02267a5-original-pull-secret podName:504ae0de-dca4-48aa-964c-b457c02267a5 nodeName:}" failed. No retries permitted until 2026-09-17 00:09:56.246825916 +0000 UTC m=+9.105319033 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/504ae0de-dca4-48aa-964c-b457c02267a5-original-pull-secret") pod "global-pull-secret-syncer-ch5w2" (UID: "504ae0de-dca4-48aa-964c-b457c02267a5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 00:09:52.247511 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:52.247298    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:09:52.247511 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:52.247354    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs podName:b956a62e-c4ac-4e1d-aeb0-3f7907a0a702 nodeName:}" failed. No retries permitted until 2026-09-17 00:09:56.247336812 +0000 UTC m=+9.105829930 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs") pod "network-metrics-daemon-xbhz5" (UID: "b956a62e-c4ac-4e1d-aeb0-3f7907a0a702") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:09:52.247511 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:52.247476    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 00:09:52.247511 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:52.247504    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 00:09:52.247740 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:52.247519    2577 projected.go:196] Error preparing data for projected volume kube-api-access-gs5sk for pod openshift-network-diagnostics/network-check-target-s8lnt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:09:52.247740 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:52.247558    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5f1a947-783b-4a35-a70b-b850883fef06-kube-api-access-gs5sk podName:f5f1a947-783b-4a35-a70b-b850883fef06 nodeName:}" failed. No retries permitted until 2026-09-17 00:09:56.247545825 +0000 UTC m=+9.106038944 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gs5sk" (UniqueName: "kubernetes.io/projected/f5f1a947-783b-4a35-a70b-b850883fef06-kube-api-access-gs5sk") pod "network-check-target-s8lnt" (UID: "f5f1a947-783b-4a35-a70b-b850883fef06") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:09:52.601733 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:52.601625    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:09:52.601733 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:52.601651    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:09:52.602033 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:52.601775    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch5w2" podUID="504ae0de-dca4-48aa-964c-b457c02267a5"
Sep 17 00:09:52.602033 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:52.601863    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbhz5" podUID="b956a62e-c4ac-4e1d-aeb0-3f7907a0a702"
Sep 17 00:09:52.602033 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:52.601934    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:09:52.602033 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:52.602011    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8lnt" podUID="f5f1a947-783b-4a35-a70b-b850883fef06"
Sep 17 00:09:54.601794 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:54.601602    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:09:54.601794 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:54.601631    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:09:54.601794 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:54.601619    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:09:54.601794 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:54.601758    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch5w2" podUID="504ae0de-dca4-48aa-964c-b457c02267a5"
Sep 17 00:09:54.602462 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:54.601852    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8lnt" podUID="f5f1a947-783b-4a35-a70b-b850883fef06"
Sep 17 00:09:54.602462 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:54.601930    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbhz5" podUID="b956a62e-c4ac-4e1d-aeb0-3f7907a0a702"
Sep 17 00:09:56.284082 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:56.284045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/504ae0de-dca4-48aa-964c-b457c02267a5-original-pull-secret\") pod \"global-pull-secret-syncer-ch5w2\" (UID: \"504ae0de-dca4-48aa-964c-b457c02267a5\") " pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:09:56.284691 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:56.284118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs\") pod \"network-metrics-daemon-xbhz5\" (UID: \"b956a62e-c4ac-4e1d-aeb0-3f7907a0a702\") " pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:09:56.284691 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:56.284156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs5sk\" (UniqueName: \"kubernetes.io/projected/f5f1a947-783b-4a35-a70b-b850883fef06-kube-api-access-gs5sk\") pod \"network-check-target-s8lnt\" (UID: \"f5f1a947-783b-4a35-a70b-b850883fef06\") " pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:09:56.284691 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:56.284243    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 00:09:56.284691 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:56.284275    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 00:09:56.284691 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:56.284291    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 00:09:56.284691 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:56.284304    2577 projected.go:196] Error preparing data for projected volume kube-api-access-gs5sk for pod openshift-network-diagnostics/network-check-target-s8lnt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:09:56.284691 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:56.284312    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/504ae0de-dca4-48aa-964c-b457c02267a5-original-pull-secret podName:504ae0de-dca4-48aa-964c-b457c02267a5 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:04.284292493 +0000 UTC m=+17.142785626 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/504ae0de-dca4-48aa-964c-b457c02267a5-original-pull-secret") pod "global-pull-secret-syncer-ch5w2" (UID: "504ae0de-dca4-48aa-964c-b457c02267a5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 00:09:56.284691 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:56.284340    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5f1a947-783b-4a35-a70b-b850883fef06-kube-api-access-gs5sk podName:f5f1a947-783b-4a35-a70b-b850883fef06 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:04.284329031 +0000 UTC m=+17.142822146 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gs5sk" (UniqueName: "kubernetes.io/projected/f5f1a947-783b-4a35-a70b-b850883fef06-kube-api-access-gs5sk") pod "network-check-target-s8lnt" (UID: "f5f1a947-783b-4a35-a70b-b850883fef06") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:09:56.284691 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:56.284399    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:09:56.284691 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:56.284432    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs podName:b956a62e-c4ac-4e1d-aeb0-3f7907a0a702 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:04.284420448 +0000 UTC m=+17.142913563 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs") pod "network-metrics-daemon-xbhz5" (UID: "b956a62e-c4ac-4e1d-aeb0-3f7907a0a702") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:09:56.602081 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:56.601962    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:09:56.602259 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:56.602108    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8lnt" podUID="f5f1a947-783b-4a35-a70b-b850883fef06"
Sep 17 00:09:56.602259 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:56.601962    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:09:56.602259 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:56.602233    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbhz5" podUID="b956a62e-c4ac-4e1d-aeb0-3f7907a0a702"
Sep 17 00:09:56.602259 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:56.601962    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:09:56.602512 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:56.602318    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch5w2" podUID="504ae0de-dca4-48aa-964c-b457c02267a5"
Sep 17 00:09:58.601251 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:58.601174    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:09:58.601251 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:58.601209    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:09:58.601845 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:09:58.601174    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:09:58.601845 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:58.601304    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8lnt" podUID="f5f1a947-783b-4a35-a70b-b850883fef06"
Sep 17 00:09:58.601845 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:58.601415    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch5w2" podUID="504ae0de-dca4-48aa-964c-b457c02267a5"
Sep 17 00:09:58.601845 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:09:58.601524    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbhz5" podUID="b956a62e-c4ac-4e1d-aeb0-3f7907a0a702"
Sep 17 00:10:00.602109 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:00.602068    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:10:00.602518 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:00.602113    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:10:00.602518 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:00.602113    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:10:00.602518 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:00.602180    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8lnt" podUID="f5f1a947-783b-4a35-a70b-b850883fef06"
Sep 17 00:10:00.605425 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:00.602674    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbhz5" podUID="b956a62e-c4ac-4e1d-aeb0-3f7907a0a702"
Sep 17 00:10:00.605425 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:00.602821    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch5w2" podUID="504ae0de-dca4-48aa-964c-b457c02267a5"
Sep 17 00:10:02.601482 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:02.601447    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:10:02.601930 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:02.601448    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:10:02.601930 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:02.601574    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8lnt" podUID="f5f1a947-783b-4a35-a70b-b850883fef06"
Sep 17 00:10:02.601930 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:02.601451    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:10:02.601930 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:02.601666    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch5w2" podUID="504ae0de-dca4-48aa-964c-b457c02267a5"
Sep 17 00:10:02.601930 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:02.601714    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbhz5" podUID="b956a62e-c4ac-4e1d-aeb0-3f7907a0a702"
Sep 17 00:10:04.347593 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:04.347551    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs\") pod \"network-metrics-daemon-xbhz5\" (UID: \"b956a62e-c4ac-4e1d-aeb0-3f7907a0a702\") " pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:10:04.348081 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:04.347608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs5sk\" (UniqueName: \"kubernetes.io/projected/f5f1a947-783b-4a35-a70b-b850883fef06-kube-api-access-gs5sk\") pod \"network-check-target-s8lnt\" (UID: \"f5f1a947-783b-4a35-a70b-b850883fef06\") " pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:10:04.348081 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:04.347651    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/504ae0de-dca4-48aa-964c-b457c02267a5-original-pull-secret\") pod \"global-pull-secret-syncer-ch5w2\" (UID: \"504ae0de-dca4-48aa-964c-b457c02267a5\") " pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:10:04.348081 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:04.347726    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:10:04.348081 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:04.347761    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 00:10:04.348081 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:04.347796    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 00:10:04.348081 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:04.347808    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs podName:b956a62e-c4ac-4e1d-aeb0-3f7907a0a702 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:20.347786946 +0000 UTC m=+33.206280081 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs") pod "network-metrics-daemon-xbhz5" (UID: "b956a62e-c4ac-4e1d-aeb0-3f7907a0a702") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:10:04.348081 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:04.347819    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 00:10:04.348081 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:04.347828    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/504ae0de-dca4-48aa-964c-b457c02267a5-original-pull-secret podName:504ae0de-dca4-48aa-964c-b457c02267a5 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:20.347821288 +0000 UTC m=+33.206314402 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/504ae0de-dca4-48aa-964c-b457c02267a5-original-pull-secret") pod "global-pull-secret-syncer-ch5w2" (UID: "504ae0de-dca4-48aa-964c-b457c02267a5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 00:10:04.348081 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:04.347835    2577 projected.go:196] Error preparing data for projected volume kube-api-access-gs5sk for pod openshift-network-diagnostics/network-check-target-s8lnt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:10:04.348081 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:04.347879    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5f1a947-783b-4a35-a70b-b850883fef06-kube-api-access-gs5sk podName:f5f1a947-783b-4a35-a70b-b850883fef06 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:20.347867925 +0000 UTC m=+33.206361039 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gs5sk" (UniqueName: "kubernetes.io/projected/f5f1a947-783b-4a35-a70b-b850883fef06-kube-api-access-gs5sk") pod "network-check-target-s8lnt" (UID: "f5f1a947-783b-4a35-a70b-b850883fef06") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:10:04.602001 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:04.601918    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:10:04.602172 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:04.601918    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:10:04.602172 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:04.602059    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbhz5" podUID="b956a62e-c4ac-4e1d-aeb0-3f7907a0a702"
Sep 17 00:10:04.602172 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:04.601929    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:10:04.602311 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:04.602119    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch5w2" podUID="504ae0de-dca4-48aa-964c-b457c02267a5"
Sep 17 00:10:04.602356 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:04.602291    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8lnt" podUID="f5f1a947-783b-4a35-a70b-b850883fef06"
Sep 17 00:10:06.601636 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:06.601596    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:10:06.602089 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:06.601596    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:10:06.602089 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:06.601740    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch5w2" podUID="504ae0de-dca4-48aa-964c-b457c02267a5"
Sep 17 00:10:06.602089 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:06.601825    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8lnt" podUID="f5f1a947-783b-4a35-a70b-b850883fef06"
Sep 17 00:10:06.602089 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:06.601596    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:10:06.602089 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:06.601944    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbhz5" podUID="b956a62e-c4ac-4e1d-aeb0-3f7907a0a702"
Sep 17 00:10:08.601801 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:08.601765    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:10:08.602258 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:08.601882    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:10:08.602258 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:08.602003    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch5w2" podUID="504ae0de-dca4-48aa-964c-b457c02267a5"
Sep 17 00:10:08.602258 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:08.602174    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbhz5" podUID="b956a62e-c4ac-4e1d-aeb0-3f7907a0a702"
Sep 17 00:10:08.602258 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:08.602203    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:10:08.602437 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:08.602283    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8lnt" podUID="f5f1a947-783b-4a35-a70b-b850883fef06"
Sep 17 00:10:09.814040 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.813640    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vml9z" event={"ID":"924b0fd5-7d3d-49cd-9ecb-bd4430ecec27","Type":"ContainerStarted","Data":"b1d57fcd2e0c78030ea3914195c98b2283e7264269e529164bdd0b6a7d92c137"}
Sep 17 00:10:09.815183 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.815152    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-w8ps8" event={"ID":"c956d3aa-d437-4b53-afc6-56127f4e1688","Type":"ContainerStarted","Data":"22935f606ec4fc713b62b6472f564fe39c0083b344aaf8d49b685bfc5bc4a30e"}
Sep 17 00:10:09.816475 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.816451    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-27.ec2.internal" event={"ID":"fbf1e8714548f742e80d275d4df6599c","Type":"ContainerStarted","Data":"ee755b68a7e70071fab0f8766630d986b6a5de8fe5d3a9eecf8d4bd96b8fba01"}
Sep 17 00:10:09.817968 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.817949    2577 generic.go:356] "Generic (PLEG): container finished" podID="ff3d9d4f106049a4c07b46e4d77c855d" containerID="b0b48eca2fd8474ad76ce1f725e302da53e8a4abb6a22872e11db9b04e496ad9" exitCode=0
Sep 17 00:10:09.818052 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.818008    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-27.ec2.internal" event={"ID":"ff3d9d4f106049a4c07b46e4d77c855d","Type":"ContainerDied","Data":"b0b48eca2fd8474ad76ce1f725e302da53e8a4abb6a22872e11db9b04e496ad9"}
Sep 17 00:10:09.819544 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.819521    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k2t7f" event={"ID":"0a948860-0f14-46ec-afab-e23492108eaa","Type":"ContainerStarted","Data":"79627a21b48ae92d8df593cc6347843549c8951465a5ae90c9604954e9343dd6"}
Sep 17 00:10:09.820979 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.820962    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl" event={"ID":"0741efde-4990-4669-9939-d5637d40f0e6","Type":"ContainerStarted","Data":"7e3f26ee40406323acb663aab82f5a5618e27e84beaab26a9d7fefc4fd627955"}
Sep 17 00:10:09.822243 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.822224    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rg8zr" event={"ID":"e6570468-fe3a-4512-8bda-9880edc3ff96","Type":"ContainerStarted","Data":"f5bab0052eddcdacd7fa72b2c1a1a27bcd38f26e43094c7c70a08a16cf51293d"}
Sep 17 00:10:09.824717 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.824700    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/ovn-acl-logging/0.log"
Sep 17 00:10:09.824996 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.824981    2577 generic.go:356] "Generic (PLEG): container finished" podID="d261c1cf-76d0-468c-a122-1c2b359f3765" containerID="a3dab3ecc0cb1e21ebc1157c7cb617744d567d5faaa6afff05b2e9e6ed6b1183" exitCode=1
Sep 17 00:10:09.825066 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.825047    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bz685" event={"ID":"d261c1cf-76d0-468c-a122-1c2b359f3765","Type":"ContainerStarted","Data":"653dd2a5febc0a6429c3eae639a0d809918bd060973ec89c5a42c248b945c270"}
Sep 17 00:10:09.825138 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.825123    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bz685" event={"ID":"d261c1cf-76d0-468c-a122-1c2b359f3765","Type":"ContainerStarted","Data":"4b9bc6f298d5f6b28ed346283b786a8e37bd5a68701c8c2a70f5762a5d9480a7"}
Sep 17 00:10:09.825192 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.825180    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bz685" event={"ID":"d261c1cf-76d0-468c-a122-1c2b359f3765","Type":"ContainerStarted","Data":"2d8a45a616af636afe700b0925cc49f02ce396bc420941a85b8ca48aa2fa8250"}
Sep 17 00:10:09.825240 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.825229    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bz685" event={"ID":"d261c1cf-76d0-468c-a122-1c2b359f3765","Type":"ContainerStarted","Data":"3aeafafce81df8106b2b72e6edf3448e6cd97457560619af7534f6678bec9e3f"}
Sep 17 00:10:09.825291 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.825262    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bz685" event={"ID":"d261c1cf-76d0-468c-a122-1c2b359f3765","Type":"ContainerDied","Data":"a3dab3ecc0cb1e21ebc1157c7cb617744d567d5faaa6afff05b2e9e6ed6b1183"}
Sep 17 00:10:09.825346 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.825292    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bz685" event={"ID":"d261c1cf-76d0-468c-a122-1c2b359f3765","Type":"ContainerStarted","Data":"1a2b2399747cbdf55006d5f0acecf1364b01a1ddec92c82adee457c6466bad24"}
Sep 17 00:10:09.826405 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.826386    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9xxfq" event={"ID":"3e97bfc2-d46a-4384-bc1a-a1a2d37afe75","Type":"ContainerStarted","Data":"b855488c98f1a68526c0c7afe56de206069cea10cd3f37f4a0a50874f4a3255c"}
Sep 17 00:10:09.827787 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.827767    2577 generic.go:356] "Generic (PLEG): container finished" podID="30b22991-ec04-4b85-9b9d-87959cfa30d1" containerID="11fd778453f3b3b0dd09dfa2786cd07c4b951a893c62e4655b18cb32777c4968" exitCode=0
Sep 17 00:10:09.827864 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.827798    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4cmx" event={"ID":"30b22991-ec04-4b85-9b9d-87959cfa30d1","Type":"ContainerDied","Data":"11fd778453f3b3b0dd09dfa2786cd07c4b951a893c62e4655b18cb32777c4968"}
Sep 17 00:10:09.842836 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.842795    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-27.ec2.internal" podStartSLOduration=20.842782575 podStartE2EDuration="20.842782575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:10:09.842555845 +0000 UTC m=+22.701048981" watchObservedRunningTime="2026-09-17 00:10:09.842782575 +0000 UTC m=+22.701275709"
Sep 17 00:10:09.842909 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.842858    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vml9z" podStartSLOduration=2.055791329 podStartE2EDuration="21.842853722s" totalImagesPullingTime="19.787062393s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:10:09.830850911 +0000 UTC m=+22.689344047" watchObservedRunningTime="2026-09-17 00:10:09.842853722 +0000 UTC m=+22.701346857"
Sep 17 00:10:09.858617 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.858576    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-w8ps8" podStartSLOduration=9.409191512 podStartE2EDuration="21.858565656s" totalImagesPullingTime="12.449374144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:10:09.858320413 +0000 UTC m=+22.716813550" watchObservedRunningTime="2026-09-17 00:10:09.858565656 +0000 UTC m=+22.717058790"
Sep 17 00:10:09.872230 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.872184    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rg8zr" podStartSLOduration=2.127826033 podStartE2EDuration="21.87216697s" totalImagesPullingTime="19.744340937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:10:09.871816933 +0000 UTC m=+22.730310069" watchObservedRunningTime="2026-09-17 00:10:09.87216697 +0000 UTC m=+22.730660105"
Sep 17 00:10:09.909820 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.909780    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9xxfq" podStartSLOduration=2.190849485 podStartE2EDuration="21.909764405s" totalImagesPullingTime="19.71891492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:10:09.909547147 +0000 UTC m=+22.768040282" watchObservedRunningTime="2026-09-17 00:10:09.909764405 +0000 UTC m=+22.768257541"
Sep 17 00:10:09.936872 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:09.936825    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-k2t7f" podStartSLOduration=1.974618515 podStartE2EDuration="21.93681215s" totalImagesPullingTime="19.962193635s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:10:09.936629304 +0000 UTC m=+22.795122433" watchObservedRunningTime="2026-09-17 00:10:09.93681215 +0000 UTC m=+22.795305285"
Sep 17 00:10:10.601177 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:10.601140    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:10:10.601386 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:10.601271    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch5w2" podUID="504ae0de-dca4-48aa-964c-b457c02267a5"
Sep 17 00:10:10.601386 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:10.601372    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:10:10.601540 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:10.601487    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbhz5" podUID="b956a62e-c4ac-4e1d-aeb0-3f7907a0a702"
Sep 17 00:10:10.601596 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:10.601553    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:10:10.601651 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:10.601620    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8lnt" podUID="f5f1a947-783b-4a35-a70b-b850883fef06"
Sep 17 00:10:10.812771 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:10.812740    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-w8ps8"
Sep 17 00:10:10.813361 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:10.813340    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-w8ps8"
Sep 17 00:10:10.833953 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:10.833917    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-27.ec2.internal" event={"ID":"ff3d9d4f106049a4c07b46e4d77c855d","Type":"ContainerStarted","Data":"c49eb82a6ddf1a3182232e8f18c803a190bca0e44ce70c9c1369b389abb1dcc9"}
Sep 17 00:10:10.835474 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:10.835446    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4bds9" event={"ID":"5685bd44-e964-4176-ae6e-479bcee01cbb","Type":"ContainerStarted","Data":"b7866be3304d340f703d4f14c3eb03bced29f6b7867d007d3d5ee6d7a1c8b232"}
Sep 17 00:10:10.836118 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:10.836099    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-w8ps8"
Sep 17 00:10:10.836665 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:10.836652    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-w8ps8"
Sep 17 00:10:10.847191 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:10.847147    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-27.ec2.internal" podStartSLOduration=21.84713047 podStartE2EDuration="21.84713047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:10:10.84661505 +0000 UTC m=+23.705108184" watchObservedRunningTime="2026-09-17 00:10:10.84713047 +0000 UTC m=+23.705623603"
Sep 17 00:10:10.873300 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:10.873220    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4bds9" podStartSLOduration=3.0906424279999998 podStartE2EDuration="22.873206928s" totalImagesPullingTime="19.7825645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:10:10.872889149 +0000 UTC m=+23.731382284" watchObservedRunningTime="2026-09-17 00:10:10.873206928 +0000 UTC m=+23.731700062"
Sep 17 00:10:11.073916 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:11.073755    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 00:10:11.565928 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:11.565780    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T00:10:11.07390463Z","UUID":"f49cba5b-e958-4aaf-b729-7ac56b67c9a9","Handler":null,"Name":"","Endpoint":""}
Sep 17 00:10:11.567742 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:11.567723    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 00:10:11.567849 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:11.567751    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 00:10:11.841387 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:11.841293    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl" event={"ID":"0741efde-4990-4669-9939-d5637d40f0e6","Type":"ContainerStarted","Data":"f18f67be1f38079078ce4c8e338a1f15a85e3ac74ad00641d0fa0af9d8302a08"}
Sep 17 00:10:11.845908 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:11.845882    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/ovn-acl-logging/0.log"
Sep 17 00:10:11.846446 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:11.846362    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bz685" event={"ID":"d261c1cf-76d0-468c-a122-1c2b359f3765","Type":"ContainerStarted","Data":"461c99417609587ab38a7611f32abcea772f58781db3a9727c2657917038c41f"}
Sep 17 00:10:12.602180 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:12.602147    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:10:12.602375 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:12.602147    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:10:12.602375 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:12.602272    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbhz5" podUID="b956a62e-c4ac-4e1d-aeb0-3f7907a0a702"
Sep 17 00:10:12.602375 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:12.602150    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:10:12.602375 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:12.602336    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8lnt" podUID="f5f1a947-783b-4a35-a70b-b850883fef06"
Sep 17 00:10:12.602619 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:12.602452    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch5w2" podUID="504ae0de-dca4-48aa-964c-b457c02267a5"
Sep 17 00:10:12.851159 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:12.851078    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl" event={"ID":"0741efde-4990-4669-9939-d5637d40f0e6","Type":"ContainerStarted","Data":"e2d182e1d4eff87a4c3a5b60d66178326f345ab32db5d57f8b2c16b688488db6"}
Sep 17 00:10:12.868240 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:12.868185    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g5bpl" podStartSLOduration=2.159700471 podStartE2EDuration="24.868169555s" totalImagesPullingTime="22.708469084s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:09:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:10:12.868131 +0000 UTC m=+25.726624147" watchObservedRunningTime="2026-09-17 00:10:12.868169555 +0000 UTC m=+25.726662672"
Sep 17 00:10:13.858237 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:13.858209    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/ovn-acl-logging/0.log"
Sep 17 00:10:13.858787 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:13.858759    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bz685" event={"ID":"d261c1cf-76d0-468c-a122-1c2b359f3765","Type":"ContainerStarted","Data":"567953bade04a041d4cef5dcaf02b65d0565545b9fec67fb2c2926b2f6d0addb"}
Sep 17 00:10:13.859333 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:13.859309    2577 scope.go:117] "RemoveContainer" containerID="a3dab3ecc0cb1e21ebc1157c7cb617744d567d5faaa6afff05b2e9e6ed6b1183"
Sep 17 00:10:14.601951 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:14.601769    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:10:14.602112 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:14.601770    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:10:14.602112 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:14.601781    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:10:14.602112 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:14.602078    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch5w2" podUID="504ae0de-dca4-48aa-964c-b457c02267a5"
Sep 17 00:10:14.602274 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:14.602144    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbhz5" podUID="b956a62e-c4ac-4e1d-aeb0-3f7907a0a702"
Sep 17 00:10:14.602274 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:14.602201    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8lnt" podUID="f5f1a947-783b-4a35-a70b-b850883fef06"
Sep 17 00:10:15.636180 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:15.636152    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ch5w2"]
Sep 17 00:10:15.636774 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:15.636261    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:10:15.636774 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:15.636380    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch5w2" podUID="504ae0de-dca4-48aa-964c-b457c02267a5"
Sep 17 00:10:15.640503 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:15.640470    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xbhz5"]
Sep 17 00:10:15.640665 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:15.640649    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:10:15.643256 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:15.640798    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbhz5" podUID="b956a62e-c4ac-4e1d-aeb0-3f7907a0a702"
Sep 17 00:10:15.643256 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:15.640922    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s8lnt"]
Sep 17 00:10:15.643256 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:15.641026    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:10:15.643256 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:15.641135    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8lnt" podUID="f5f1a947-783b-4a35-a70b-b850883fef06"
Sep 17 00:10:15.865449 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:15.865395    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4cmx" event={"ID":"30b22991-ec04-4b85-9b9d-87959cfa30d1","Type":"ContainerStarted","Data":"a395b96675fd72188ccd2175feb27b43ad0d6738e74bdf19e2cf665f8637352f"}
Sep 17 00:10:15.869489 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:15.869466    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/ovn-acl-logging/0.log"
Sep 17 00:10:15.869838 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:15.869815    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bz685" event={"ID":"d261c1cf-76d0-468c-a122-1c2b359f3765","Type":"ContainerStarted","Data":"0e5c47c6e87ff61d61adffc17514f54f8c9f573f8ed9367946038c93f6efce53"}
Sep 17 00:10:15.870131 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:15.870104    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:10:15.870235 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:15.870192    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:10:15.870300 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:15.870242    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:10:15.884967 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:15.884947    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:10:15.885058 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:15.885040    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:10:15.907317 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:15.907273    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bz685" podStartSLOduration=8.080266238 podStartE2EDuration="27.907258853s" totalImagesPullingTime="19.826992615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:10:15.905693707 +0000 UTC m=+28.764186866" watchObservedRunningTime="2026-09-17 00:10:15.907258853 +0000 UTC m=+28.765751987"
Sep 17 00:10:16.873844 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:16.873810    2577 generic.go:356] "Generic (PLEG): container finished" podID="30b22991-ec04-4b85-9b9d-87959cfa30d1" containerID="a395b96675fd72188ccd2175feb27b43ad0d6738e74bdf19e2cf665f8637352f" exitCode=0
Sep 17 00:10:16.874373 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:16.873891    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4cmx" event={"ID":"30b22991-ec04-4b85-9b9d-87959cfa30d1","Type":"ContainerDied","Data":"a395b96675fd72188ccd2175feb27b43ad0d6738e74bdf19e2cf665f8637352f"}
Sep 17 00:10:17.601982 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:17.601952    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:10:17.602145 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:17.602064    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbhz5" podUID="b956a62e-c4ac-4e1d-aeb0-3f7907a0a702"
Sep 17 00:10:17.602145 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:17.602088    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:10:17.602253 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:17.602195    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8lnt" podUID="f5f1a947-783b-4a35-a70b-b850883fef06"
Sep 17 00:10:17.602253 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:17.602227    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:10:17.602321 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:17.602298    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch5w2" podUID="504ae0de-dca4-48aa-964c-b457c02267a5"
Sep 17 00:10:18.880449 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:18.880406    2577 generic.go:356] "Generic (PLEG): container finished" podID="30b22991-ec04-4b85-9b9d-87959cfa30d1" containerID="effef1ee0627a8cde0bd5523c4467f5959f6e02018edc7ab1a26b6b8e77f453d" exitCode=0
Sep 17 00:10:18.880449 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:18.880465    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4cmx" event={"ID":"30b22991-ec04-4b85-9b9d-87959cfa30d1","Type":"ContainerDied","Data":"effef1ee0627a8cde0bd5523c4467f5959f6e02018edc7ab1a26b6b8e77f453d"}
Sep 17 00:10:19.601462 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:19.601408    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:10:19.601640 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:19.601408    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:10:19.601640 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:19.601568    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbhz5" podUID="b956a62e-c4ac-4e1d-aeb0-3f7907a0a702"
Sep 17 00:10:19.601640 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:19.601407    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:10:19.601640 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:19.601612    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch5w2" podUID="504ae0de-dca4-48aa-964c-b457c02267a5"
Sep 17 00:10:19.601885 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:19.601680    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8lnt" podUID="f5f1a947-783b-4a35-a70b-b850883fef06"
Sep 17 00:10:20.366572 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:20.366500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs\") pod \"network-metrics-daemon-xbhz5\" (UID: \"b956a62e-c4ac-4e1d-aeb0-3f7907a0a702\") " pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:10:20.366572 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:20.366541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs5sk\" (UniqueName: \"kubernetes.io/projected/f5f1a947-783b-4a35-a70b-b850883fef06-kube-api-access-gs5sk\") pod \"network-check-target-s8lnt\" (UID: \"f5f1a947-783b-4a35-a70b-b850883fef06\") " pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:10:20.366905 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:20.366616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/504ae0de-dca4-48aa-964c-b457c02267a5-original-pull-secret\") pod \"global-pull-secret-syncer-ch5w2\" (UID: \"504ae0de-dca4-48aa-964c-b457c02267a5\") " pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:10:20.366905 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:20.366648    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 00:10:20.366905 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:20.366664    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 00:10:20.366905 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:20.366672    2577 projected.go:196] Error preparing data for projected volume kube-api-access-gs5sk for pod openshift-network-diagnostics/network-check-target-s8lnt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:10:20.366905 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:20.366699    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 00:10:20.366905 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:20.366715    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5f1a947-783b-4a35-a70b-b850883fef06-kube-api-access-gs5sk podName:f5f1a947-783b-4a35-a70b-b850883fef06 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:52.366700153 +0000 UTC m=+65.225193283 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gs5sk" (UniqueName: "kubernetes.io/projected/f5f1a947-783b-4a35-a70b-b850883fef06-kube-api-access-gs5sk") pod "network-check-target-s8lnt" (UID: "f5f1a947-783b-4a35-a70b-b850883fef06") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 00:10:20.366905 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:20.366737    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/504ae0de-dca4-48aa-964c-b457c02267a5-original-pull-secret podName:504ae0de-dca4-48aa-964c-b457c02267a5 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:52.366725 +0000 UTC m=+65.225218114 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/504ae0de-dca4-48aa-964c-b457c02267a5-original-pull-secret") pod "global-pull-secret-syncer-ch5w2" (UID: "504ae0de-dca4-48aa-964c-b457c02267a5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 00:10:20.366905 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:20.366756    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:10:20.366905 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:20.366785    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs podName:b956a62e-c4ac-4e1d-aeb0-3f7907a0a702 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:52.366777533 +0000 UTC m=+65.225270646 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs") pod "network-metrics-daemon-xbhz5" (UID: "b956a62e-c4ac-4e1d-aeb0-3f7907a0a702") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 00:10:20.886853 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:20.886820    2577 generic.go:356] "Generic (PLEG): container finished" podID="30b22991-ec04-4b85-9b9d-87959cfa30d1" containerID="4bdf7619c8d6ce526efbd263b164a45ea0c0fa5e4bdd267da053998c20ff3617" exitCode=0
Sep 17 00:10:20.886853 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:20.886859    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4cmx" event={"ID":"30b22991-ec04-4b85-9b9d-87959cfa30d1","Type":"ContainerDied","Data":"4bdf7619c8d6ce526efbd263b164a45ea0c0fa5e4bdd267da053998c20ff3617"}
Sep 17 00:10:21.604242 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:21.604205    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:10:21.604715 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:21.604207    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:10:21.604715 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:21.604324    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch5w2" podUID="504ae0de-dca4-48aa-964c-b457c02267a5"
Sep 17 00:10:21.604715 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:21.604207    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:10:21.604715 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:21.604379    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbhz5" podUID="b956a62e-c4ac-4e1d-aeb0-3f7907a0a702"
Sep 17 00:10:21.604715 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:21.604452    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s8lnt" podUID="f5f1a947-783b-4a35-a70b-b850883fef06"
Sep 17 00:10:22.490903 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.490875    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-27.ec2.internal" event="NodeReady"
Sep 17 00:10:22.491090 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.491001    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 00:10:22.522523 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.522487    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x"]
Sep 17 00:10:22.543280 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.543252    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5957bbc947-plqh5"]
Sep 17 00:10:22.544939 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.544249    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x"
Sep 17 00:10:22.546960 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.546937    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-8njzj\""
Sep 17 00:10:22.547270 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.547106    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 00:10:22.547381 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.547317    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 00:10:22.561153 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.561128    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x"]
Sep 17 00:10:22.561277 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.561169    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-twmbk"]
Sep 17 00:10:22.561277 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.561262    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:22.564005 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.563732    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 00:10:22.564163 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.564141    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 00:10:22.564372 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.564144    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 00:10:22.564457 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.564355    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j25ch\""
Sep 17 00:10:22.572299 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.572278    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 00:10:22.583280 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.583257    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5957bbc947-plqh5"]
Sep 17 00:10:22.583397 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.583292    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zhxfj"]
Sep 17 00:10:22.583475 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.583390    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-twmbk"
Sep 17 00:10:22.586584 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.586558    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 00:10:22.586689 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.586658    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 00:10:22.586689 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.586681    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 00:10:22.586785 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.586572    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cbrk8\""
Sep 17 00:10:22.603352 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.603326    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-twmbk"]
Sep 17 00:10:22.603483 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.603363    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zhxfj"]
Sep 17 00:10:22.603483 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.603471    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:10:22.606051 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.606022    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 00:10:22.606400 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.606097    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d74lr\""
Sep 17 00:10:22.606400 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.606157    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 00:10:22.682780 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.682740    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e823b26-89d1-4a1d-95b0-1d0002258005-installation-pull-secrets\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:22.682780 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.682782    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls\") pod \"dns-default-zhxfj\" (UID: \"57f31c58-e354-4d56-8b7e-c55015c26176\") " pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:10:22.682991 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.682812    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-799bd\" (UniqueName: \"kubernetes.io/projected/57f31c58-e354-4d56-8b7e-c55015c26176-kube-api-access-799bd\") pod \"dns-default-zhxfj\" (UID: \"57f31c58-e354-4d56-8b7e-c55015c26176\") " pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:10:22.682991 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.682876    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-858l4\" (UniqueName: \"kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-kube-api-access-858l4\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:22.682991 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.682927    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e823b26-89d1-4a1d-95b0-1d0002258005-ca-trust-extracted\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:22.682991 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.682948    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccc9a7ce-9b6a-40f4-9d5e-388c3e892f11-cert\") pod \"ingress-canary-twmbk\" (UID: \"ccc9a7ce-9b6a-40f4-9d5e-388c3e892f11\") " pod="openshift-ingress-canary/ingress-canary-twmbk"
Sep 17 00:10:22.682991 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.682972    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhcnb\" (UniqueName: \"kubernetes.io/projected/ccc9a7ce-9b6a-40f4-9d5e-388c3e892f11-kube-api-access-nhcnb\") pod \"ingress-canary-twmbk\" (UID: \"ccc9a7ce-9b6a-40f4-9d5e-388c3e892f11\") " pod="openshift-ingress-canary/ingress-canary-twmbk"
Sep 17 00:10:22.683163 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.683009    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b91229a2-6f57-4a69-af2c-8cd92b0459f0-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-cch7x\" (UID: \"b91229a2-6f57-4a69-af2c-8cd92b0459f0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x"
Sep 17 00:10:22.683163 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.683034    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57f31c58-e354-4d56-8b7e-c55015c26176-config-volume\") pod \"dns-default-zhxfj\" (UID: \"57f31c58-e354-4d56-8b7e-c55015c26176\") " pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:10:22.683163 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.683078    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e823b26-89d1-4a1d-95b0-1d0002258005-trusted-ca\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:22.683163 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.683130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e823b26-89d1-4a1d-95b0-1d0002258005-image-registry-private-configuration\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:22.683163 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.683149    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-certificates\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:22.683329 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.683166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-bound-sa-token\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:22.683329 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.683185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f31c58-e354-4d56-8b7e-c55015c26176-tmp-dir\") pod \"dns-default-zhxfj\" (UID: \"57f31c58-e354-4d56-8b7e-c55015c26176\") " pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:10:22.683329 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.683252    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:22.683329 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.683295    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-cch7x\" (UID: \"b91229a2-6f57-4a69-af2c-8cd92b0459f0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x"
Sep 17 00:10:22.783666 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.783632    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57f31c58-e354-4d56-8b7e-c55015c26176-config-volume\") pod \"dns-default-zhxfj\" (UID: \"57f31c58-e354-4d56-8b7e-c55015c26176\") " pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:10:22.783666 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.783670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e823b26-89d1-4a1d-95b0-1d0002258005-trusted-ca\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:22.783892 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.783716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e823b26-89d1-4a1d-95b0-1d0002258005-image-registry-private-configuration\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:22.783892 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.783734    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-certificates\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:22.783892 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.783831    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-bound-sa-token\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:22.783892 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.783881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f31c58-e354-4d56-8b7e-c55015c26176-tmp-dir\") pod \"dns-default-zhxfj\" (UID: \"57f31c58-e354-4d56-8b7e-c55015c26176\") " pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:10:22.784084 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.783929    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:22.784084 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.783975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-cch7x\" (UID: \"b91229a2-6f57-4a69-af2c-8cd92b0459f0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x"
Sep 17 00:10:22.784084 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.784002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e823b26-89d1-4a1d-95b0-1d0002258005-installation-pull-secrets\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:22.784084 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.784026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls\") pod \"dns-default-zhxfj\" (UID: \"57f31c58-e354-4d56-8b7e-c55015c26176\") " pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:10:22.784084 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.784054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-799bd\" (UniqueName: \"kubernetes.io/projected/57f31c58-e354-4d56-8b7e-c55015c26176-kube-api-access-799bd\") pod \"dns-default-zhxfj\" (UID: \"57f31c58-e354-4d56-8b7e-c55015c26176\") " pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:10:22.784084 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.784082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-858l4\" (UniqueName: \"kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-kube-api-access-858l4\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:22.784353 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.784108    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e823b26-89d1-4a1d-95b0-1d0002258005-ca-trust-extracted\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:22.784353 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.784128    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccc9a7ce-9b6a-40f4-9d5e-388c3e892f11-cert\") pod \"ingress-canary-twmbk\" (UID: \"ccc9a7ce-9b6a-40f4-9d5e-388c3e892f11\") " pod="openshift-ingress-canary/ingress-canary-twmbk"
Sep 17 00:10:22.784353 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.784152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhcnb\" (UniqueName: \"kubernetes.io/projected/ccc9a7ce-9b6a-40f4-9d5e-388c3e892f11-kube-api-access-nhcnb\") pod \"ingress-canary-twmbk\" (UID: \"ccc9a7ce-9b6a-40f4-9d5e-388c3e892f11\") " pod="openshift-ingress-canary/ingress-canary-twmbk"
Sep 17 00:10:22.784353 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.784190    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b91229a2-6f57-4a69-af2c-8cd92b0459f0-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-cch7x\" (UID: \"b91229a2-6f57-4a69-af2c-8cd92b0459f0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x"
Sep 17 00:10:22.784353 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.784305    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57f31c58-e354-4d56-8b7e-c55015c26176-config-volume\") pod \"dns-default-zhxfj\" (UID: \"57f31c58-e354-4d56-8b7e-c55015c26176\") " pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:10:22.784353 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:22.784310    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 00:10:22.784669 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.784357    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-certificates\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:22.784669 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:22.784386    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls podName:57f31c58-e354-4d56-8b7e-c55015c26176 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:23.284368556 +0000 UTC m=+36.142861686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls") pod "dns-default-zhxfj" (UID: "57f31c58-e354-4d56-8b7e-c55015c26176") : secret "dns-default-metrics-tls" not found
Sep 17 00:10:22.784669 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.784603    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f31c58-e354-4d56-8b7e-c55015c26176-tmp-dir\") pod \"dns-default-zhxfj\" (UID: \"57f31c58-e354-4d56-8b7e-c55015c26176\") " pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:10:22.784827 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:22.784709    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 00:10:22.784827 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:22.784721    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 00:10:22.784827 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.784783    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e823b26-89d1-4a1d-95b0-1d0002258005-trusted-ca\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:22.784827 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:22.784796    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert podName:b91229a2-6f57-4a69-af2c-8cd92b0459f0 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:23.284779874 +0000 UTC m=+36.143273001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-cch7x" (UID: "b91229a2-6f57-4a69-af2c-8cd92b0459f0") : secret "networking-console-plugin-cert" not found
Sep 17 00:10:22.784827 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:22.784724    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5957bbc947-plqh5: secret "image-registry-tls" not found
Sep 17 00:10:22.785065 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:22.784834    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls podName:5e823b26-89d1-4a1d-95b0-1d0002258005 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:23.284825608 +0000 UTC m=+36.143318721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls") pod "image-registry-5957bbc947-plqh5" (UID: "5e823b26-89d1-4a1d-95b0-1d0002258005") : secret "image-registry-tls" not found
Sep 17 00:10:22.785065 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.784915    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b91229a2-6f57-4a69-af2c-8cd92b0459f0-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-cch7x\" (UID: \"b91229a2-6f57-4a69-af2c-8cd92b0459f0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x"
Sep 17 00:10:22.785192 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.785159    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e823b26-89d1-4a1d-95b0-1d0002258005-ca-trust-extracted\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:22.788737 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.788713    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e823b26-89d1-4a1d-95b0-1d0002258005-installation-pull-secrets\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:22.788851 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.788713    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e823b26-89d1-4a1d-95b0-1d0002258005-image-registry-private-configuration\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:22.789115 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.789090    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ccc9a7ce-9b6a-40f4-9d5e-388c3e892f11-cert\") pod \"ingress-canary-twmbk\" (UID: \"ccc9a7ce-9b6a-40f4-9d5e-388c3e892f11\") " pod="openshift-ingress-canary/ingress-canary-twmbk"
Sep 17 00:10:22.792847 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.792822    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-bound-sa-token\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:22.792992 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.792973    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-858l4\" (UniqueName: \"kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-kube-api-access-858l4\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:22.793053 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.793037    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-799bd\" (UniqueName: \"kubernetes.io/projected/57f31c58-e354-4d56-8b7e-c55015c26176-kube-api-access-799bd\") pod \"dns-default-zhxfj\" (UID: \"57f31c58-e354-4d56-8b7e-c55015c26176\") " pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:10:22.793181 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.793161    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhcnb\" (UniqueName: \"kubernetes.io/projected/ccc9a7ce-9b6a-40f4-9d5e-388c3e892f11-kube-api-access-nhcnb\") pod \"ingress-canary-twmbk\" (UID: \"ccc9a7ce-9b6a-40f4-9d5e-388c3e892f11\") " pod="openshift-ingress-canary/ingress-canary-twmbk"
Sep 17 00:10:22.893524 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:22.893448    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-twmbk"
Sep 17 00:10:23.222606 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:23.222325    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-twmbk"]
Sep 17 00:10:23.226524 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:10:23.226498    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc9a7ce_9b6a_40f4_9d5e_388c3e892f11.slice/crio-f910acfd921bb8c32b941354385d350c8bbe67352713bf7b9ec48615a579e651 WatchSource:0}: Error finding container f910acfd921bb8c32b941354385d350c8bbe67352713bf7b9ec48615a579e651: Status 404 returned error can't find the container with id f910acfd921bb8c32b941354385d350c8bbe67352713bf7b9ec48615a579e651
Sep 17 00:10:23.287328 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:23.287285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:23.287527 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:23.287342    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-cch7x\" (UID: \"b91229a2-6f57-4a69-af2c-8cd92b0459f0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x"
Sep 17 00:10:23.287527 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:23.287375    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls\") pod \"dns-default-zhxfj\" (UID: \"57f31c58-e354-4d56-8b7e-c55015c26176\") " pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:10:23.287527 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:23.287483    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 00:10:23.287527 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:23.287493    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 00:10:23.287527 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:23.287505    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5957bbc947-plqh5: secret "image-registry-tls" not found
Sep 17 00:10:23.287527 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:23.287489    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 00:10:23.287831 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:23.287554    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls podName:57f31c58-e354-4d56-8b7e-c55015c26176 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:24.287536873 +0000 UTC m=+37.146029989 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls") pod "dns-default-zhxfj" (UID: "57f31c58-e354-4d56-8b7e-c55015c26176") : secret "dns-default-metrics-tls" not found
Sep 17 00:10:23.287831 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:23.287611    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls podName:5e823b26-89d1-4a1d-95b0-1d0002258005 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:24.287604128 +0000 UTC m=+37.146097240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls") pod "image-registry-5957bbc947-plqh5" (UID: "5e823b26-89d1-4a1d-95b0-1d0002258005") : secret "image-registry-tls" not found
Sep 17 00:10:23.287831 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:23.287624    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert podName:b91229a2-6f57-4a69-af2c-8cd92b0459f0 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:24.287616628 +0000 UTC m=+37.146109741 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-cch7x" (UID: "b91229a2-6f57-4a69-af2c-8cd92b0459f0") : secret "networking-console-plugin-cert" not found
Sep 17 00:10:23.605169 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:23.605085    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:10:23.605341 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:23.605192    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:10:23.605341 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:23.605269    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:10:23.607607 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:23.607587    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-86j4t\""
Sep 17 00:10:23.607907 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:23.607612    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 00:10:23.608615 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:23.608593    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 00:10:23.608615 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:23.608609    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 00:10:23.608783 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:23.608610    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zxlvp\""
Sep 17 00:10:23.608783 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:23.608611    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 00:10:23.895737 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:23.895660    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-twmbk" event={"ID":"ccc9a7ce-9b6a-40f4-9d5e-388c3e892f11","Type":"ContainerStarted","Data":"f910acfd921bb8c32b941354385d350c8bbe67352713bf7b9ec48615a579e651"}
Sep 17 00:10:24.296025 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:24.295990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:24.296204 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:24.296051    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-cch7x\" (UID: \"b91229a2-6f57-4a69-af2c-8cd92b0459f0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x"
Sep 17 00:10:24.296204 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:24.296084    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls\") pod \"dns-default-zhxfj\" (UID: \"57f31c58-e354-4d56-8b7e-c55015c26176\") " pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:10:24.296204 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:24.296178    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 00:10:24.296204 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:24.296204    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5957bbc947-plqh5: secret "image-registry-tls" not found
Sep 17 00:10:24.296376 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:24.296223    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 00:10:24.296376 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:24.296271    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls podName:5e823b26-89d1-4a1d-95b0-1d0002258005 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:26.296249995 +0000 UTC m=+39.154743126 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls") pod "image-registry-5957bbc947-plqh5" (UID: "5e823b26-89d1-4a1d-95b0-1d0002258005") : secret "image-registry-tls" not found
Sep 17 00:10:24.296376 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:24.296290    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls podName:57f31c58-e354-4d56-8b7e-c55015c26176 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:26.296280279 +0000 UTC m=+39.154773397 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls") pod "dns-default-zhxfj" (UID: "57f31c58-e354-4d56-8b7e-c55015c26176") : secret "dns-default-metrics-tls" not found
Sep 17 00:10:24.296376 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:24.296307    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 00:10:24.296376 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:24.296344    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert podName:b91229a2-6f57-4a69-af2c-8cd92b0459f0 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:26.296327683 +0000 UTC m=+39.154820797 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-cch7x" (UID: "b91229a2-6f57-4a69-af2c-8cd92b0459f0") : secret "networking-console-plugin-cert" not found
Sep 17 00:10:26.310765 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:26.310711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:26.311322 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:26.310783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-cch7x\" (UID: \"b91229a2-6f57-4a69-af2c-8cd92b0459f0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x"
Sep 17 00:10:26.311322 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:26.310820    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls\") pod \"dns-default-zhxfj\" (UID: \"57f31c58-e354-4d56-8b7e-c55015c26176\") " pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:10:26.311322 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:26.310898    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 00:10:26.311322 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:26.310923    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5957bbc947-plqh5: secret "image-registry-tls" not found
Sep 17 00:10:26.311322 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:26.310965    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 00:10:26.311322 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:26.310973    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 00:10:26.311322 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:26.310993    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls podName:5e823b26-89d1-4a1d-95b0-1d0002258005 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:30.310970903 +0000 UTC m=+43.169464036 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls") pod "image-registry-5957bbc947-plqh5" (UID: "5e823b26-89d1-4a1d-95b0-1d0002258005") : secret "image-registry-tls" not found
Sep 17 00:10:26.311322 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:26.311025    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert podName:b91229a2-6f57-4a69-af2c-8cd92b0459f0 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:30.311008642 +0000 UTC m=+43.169501773 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-cch7x" (UID: "b91229a2-6f57-4a69-af2c-8cd92b0459f0") : secret "networking-console-plugin-cert" not found
Sep 17 00:10:26.311322 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:26.311047    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls podName:57f31c58-e354-4d56-8b7e-c55015c26176 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:30.311032027 +0000 UTC m=+43.169525141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls") pod "dns-default-zhxfj" (UID: "57f31c58-e354-4d56-8b7e-c55015c26176") : secret "dns-default-metrics-tls" not found
Sep 17 00:10:27.910624 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:27.910585    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4cmx" event={"ID":"30b22991-ec04-4b85-9b9d-87959cfa30d1","Type":"ContainerStarted","Data":"fe750e9786ddf4239b7381dc07cf38d3416fae6621ffa5380e8455f052b24b6c"}
Sep 17 00:10:27.913470 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:27.913408    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-twmbk" event={"ID":"ccc9a7ce-9b6a-40f4-9d5e-388c3e892f11","Type":"ContainerStarted","Data":"48032708187e9fc85d465a29d2fde1741bb984cdd2673a8de7b2cb0d4947cd20"}
Sep 17 00:10:27.948378 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:27.948332    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-twmbk" podStartSLOduration=1.501657987 podStartE2EDuration="5.948315365s" totalImagesPullingTime="4.446657378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:10:27.947686988 +0000 UTC m=+40.806180123" watchObservedRunningTime="2026-09-17 00:10:27.948315365 +0000 UTC m=+40.806808496"
Sep 17 00:10:28.832971 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:28.832935    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69897745dd-nhpt7"]
Sep 17 00:10:28.855316 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:28.855285    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj"]
Sep 17 00:10:28.855491 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:28.855423    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69897745dd-nhpt7"
Sep 17 00:10:28.858037 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:28.858012    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 00:10:28.858973 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:28.858954    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 00:10:28.859069 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:28.858983    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 00:10:28.859069 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:28.859017    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-sqr76\""
Sep 17 00:10:28.859069 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:28.859025    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 00:10:28.873316 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:28.873298    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69897745dd-nhpt7"]
Sep 17 00:10:28.873409 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:28.873327    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj"]
Sep 17 00:10:28.873409 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:28.873401    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj"
Sep 17 00:10:28.875543 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:28.875523    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 00:10:28.885741 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:28.885722    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474"]
Sep 17 00:10:28.911344 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:28.911318    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474"]
Sep 17 00:10:28.911713 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:28.911425    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474"
Sep 17 00:10:28.913964 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:28.913945    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 00:10:28.914100 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:28.914082    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 00:10:28.914167 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:28.914146    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 00:10:28.914471 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:28.914448    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 00:10:28.919016 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:28.918989    2577 generic.go:356] "Generic (PLEG): container finished" podID="30b22991-ec04-4b85-9b9d-87959cfa30d1" containerID="fe750e9786ddf4239b7381dc07cf38d3416fae6621ffa5380e8455f052b24b6c" exitCode=0
Sep 17 00:10:28.919110 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:28.919064    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4cmx" event={"ID":"30b22991-ec04-4b85-9b9d-87959cfa30d1","Type":"ContainerDied","Data":"fe750e9786ddf4239b7381dc07cf38d3416fae6621ffa5380e8455f052b24b6c"}
Sep 17 00:10:29.032763 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.032736    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/84a8dad6-c057-4734-a357-421d90017acd-ca\") pod \"cluster-proxy-proxy-agent-d65658c7-qq474\" (UID: \"84a8dad6-c057-4734-a357-421d90017acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474"
Sep 17 00:10:29.032763 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.032771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/84a8dad6-c057-4734-a357-421d90017acd-hub\") pod \"cluster-proxy-proxy-agent-d65658c7-qq474\" (UID: \"84a8dad6-c057-4734-a357-421d90017acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474"
Sep 17 00:10:29.032977 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.032797    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a8dad6-c057-4734-a357-421d90017acd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d65658c7-qq474\" (UID: \"84a8dad6-c057-4734-a357-421d90017acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474"
Sep 17 00:10:29.032977 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.032825    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/55925826-a2f5-4c85-9fdd-66e886482069-klusterlet-config\") pod \"klusterlet-addon-workmgr-cc45f99fb-sm5gj\" (UID: \"55925826-a2f5-4c85-9fdd-66e886482069\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj"
Sep 17 00:10:29.032977 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.032889    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qn2b\" (UniqueName: \"kubernetes.io/projected/84a8dad6-c057-4734-a357-421d90017acd-kube-api-access-4qn2b\") pod \"cluster-proxy-proxy-agent-d65658c7-qq474\" (UID: \"84a8dad6-c057-4734-a357-421d90017acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474"
Sep 17 00:10:29.033141 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.032987    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55925826-a2f5-4c85-9fdd-66e886482069-tmp\") pod \"klusterlet-addon-workmgr-cc45f99fb-sm5gj\" (UID: \"55925826-a2f5-4c85-9fdd-66e886482069\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj"
Sep 17 00:10:29.033141 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.033065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84a8dad6-c057-4734-a357-421d90017acd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d65658c7-qq474\" (UID: \"84a8dad6-c057-4734-a357-421d90017acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474"
Sep 17 00:10:29.033141 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.033116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/33131f95-e9e8-4d86-a9d8-73ff5abb5396-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69897745dd-nhpt7\" (UID: \"33131f95-e9e8-4d86-a9d8-73ff5abb5396\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69897745dd-nhpt7"
Sep 17 00:10:29.033253 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.033141    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bjxs\" (UniqueName: \"kubernetes.io/projected/33131f95-e9e8-4d86-a9d8-73ff5abb5396-kube-api-access-4bjxs\") pod \"managed-serviceaccount-addon-agent-69897745dd-nhpt7\" (UID: \"33131f95-e9e8-4d86-a9d8-73ff5abb5396\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69897745dd-nhpt7"
Sep 17 00:10:29.033253 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.033178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4xsw\" (UniqueName: \"kubernetes.io/projected/55925826-a2f5-4c85-9fdd-66e886482069-kube-api-access-p4xsw\") pod \"klusterlet-addon-workmgr-cc45f99fb-sm5gj\" (UID: \"55925826-a2f5-4c85-9fdd-66e886482069\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj"
Sep 17 00:10:29.033253 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.033195    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/84a8dad6-c057-4734-a357-421d90017acd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d65658c7-qq474\" (UID: \"84a8dad6-c057-4734-a357-421d90017acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474"
Sep 17 00:10:29.134426 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.134385    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55925826-a2f5-4c85-9fdd-66e886482069-tmp\") pod \"klusterlet-addon-workmgr-cc45f99fb-sm5gj\" (UID: \"55925826-a2f5-4c85-9fdd-66e886482069\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj"
Sep 17 00:10:29.134529 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.134446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84a8dad6-c057-4734-a357-421d90017acd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d65658c7-qq474\" (UID: \"84a8dad6-c057-4734-a357-421d90017acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474"
Sep 17 00:10:29.134529 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.134473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/33131f95-e9e8-4d86-a9d8-73ff5abb5396-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69897745dd-nhpt7\" (UID: \"33131f95-e9e8-4d86-a9d8-73ff5abb5396\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69897745dd-nhpt7"
Sep 17 00:10:29.134529 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.134490    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bjxs\" (UniqueName: \"kubernetes.io/projected/33131f95-e9e8-4d86-a9d8-73ff5abb5396-kube-api-access-4bjxs\") pod \"managed-serviceaccount-addon-agent-69897745dd-nhpt7\" (UID: \"33131f95-e9e8-4d86-a9d8-73ff5abb5396\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69897745dd-nhpt7"
Sep 17 00:10:29.134529 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.134510    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4xsw\" (UniqueName: \"kubernetes.io/projected/55925826-a2f5-4c85-9fdd-66e886482069-kube-api-access-p4xsw\") pod \"klusterlet-addon-workmgr-cc45f99fb-sm5gj\" (UID: \"55925826-a2f5-4c85-9fdd-66e886482069\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj"
Sep 17 00:10:29.134529 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.134524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/84a8dad6-c057-4734-a357-421d90017acd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d65658c7-qq474\" (UID: \"84a8dad6-c057-4734-a357-421d90017acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474"
Sep 17 00:10:29.134772 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.134551    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/84a8dad6-c057-4734-a357-421d90017acd-ca\") pod \"cluster-proxy-proxy-agent-d65658c7-qq474\" (UID: \"84a8dad6-c057-4734-a357-421d90017acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474"
Sep 17 00:10:29.134772 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.134569    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/84a8dad6-c057-4734-a357-421d90017acd-hub\") pod \"cluster-proxy-proxy-agent-d65658c7-qq474\" (UID: \"84a8dad6-c057-4734-a357-421d90017acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474"
Sep 17 00:10:29.134772 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.134595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a8dad6-c057-4734-a357-421d90017acd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d65658c7-qq474\" (UID: \"84a8dad6-c057-4734-a357-421d90017acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474"
Sep 17 00:10:29.134772 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.134624    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/55925826-a2f5-4c85-9fdd-66e886482069-klusterlet-config\") pod \"klusterlet-addon-workmgr-cc45f99fb-sm5gj\" (UID: \"55925826-a2f5-4c85-9fdd-66e886482069\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj"
Sep 17 00:10:29.134772 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.134656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qn2b\" (UniqueName: \"kubernetes.io/projected/84a8dad6-c057-4734-a357-421d90017acd-kube-api-access-4qn2b\") pod \"cluster-proxy-proxy-agent-d65658c7-qq474\" (UID: \"84a8dad6-c057-4734-a357-421d90017acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474"
Sep 17 00:10:29.135016 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.134798    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55925826-a2f5-4c85-9fdd-66e886482069-tmp\") pod \"klusterlet-addon-workmgr-cc45f99fb-sm5gj\" (UID: \"55925826-a2f5-4c85-9fdd-66e886482069\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj"
Sep 17 00:10:29.135482 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.135459    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a8dad6-c057-4734-a357-421d90017acd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d65658c7-qq474\" (UID: \"84a8dad6-c057-4734-a357-421d90017acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474"
Sep 17 00:10:29.138242 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.138220    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/84a8dad6-c057-4734-a357-421d90017acd-ca\") pod \"cluster-proxy-proxy-agent-d65658c7-qq474\" (UID: \"84a8dad6-c057-4734-a357-421d90017acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474"
Sep 17 00:10:29.138374 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.138356    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/55925826-a2f5-4c85-9fdd-66e886482069-klusterlet-config\") pod \"klusterlet-addon-workmgr-cc45f99fb-sm5gj\" (UID: \"55925826-a2f5-4c85-9fdd-66e886482069\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj"
Sep 17 00:10:29.138452 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.138432    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84a8dad6-c057-4734-a357-421d90017acd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d65658c7-qq474\" (UID: \"84a8dad6-c057-4734-a357-421d90017acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474"
Sep 17 00:10:29.138488 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.138456    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/84a8dad6-c057-4734-a357-421d90017acd-hub\") pod \"cluster-proxy-proxy-agent-d65658c7-qq474\" (UID: \"84a8dad6-c057-4734-a357-421d90017acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474"
Sep 17 00:10:29.138629 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.138610    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/33131f95-e9e8-4d86-a9d8-73ff5abb5396-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69897745dd-nhpt7\" (UID: \"33131f95-e9e8-4d86-a9d8-73ff5abb5396\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69897745dd-nhpt7"
Sep 17 00:10:29.138671 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.138654    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/84a8dad6-c057-4734-a357-421d90017acd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d65658c7-qq474\" (UID: \"84a8dad6-c057-4734-a357-421d90017acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474"
Sep 17 00:10:29.142196 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.142174    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bjxs\" (UniqueName: \"kubernetes.io/projected/33131f95-e9e8-4d86-a9d8-73ff5abb5396-kube-api-access-4bjxs\") pod \"managed-serviceaccount-addon-agent-69897745dd-nhpt7\" (UID: \"33131f95-e9e8-4d86-a9d8-73ff5abb5396\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69897745dd-nhpt7"
Sep 17 00:10:29.142359 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.142332    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qn2b\" (UniqueName: \"kubernetes.io/projected/84a8dad6-c057-4734-a357-421d90017acd-kube-api-access-4qn2b\") pod \"cluster-proxy-proxy-agent-d65658c7-qq474\" (UID: \"84a8dad6-c057-4734-a357-421d90017acd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474"
Sep 17 00:10:29.142465 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.142379    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4xsw\" (UniqueName: \"kubernetes.io/projected/55925826-a2f5-4c85-9fdd-66e886482069-kube-api-access-p4xsw\") pod \"klusterlet-addon-workmgr-cc45f99fb-sm5gj\" (UID: \"55925826-a2f5-4c85-9fdd-66e886482069\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj"
Sep 17 00:10:29.177580 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.177549    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69897745dd-nhpt7"
Sep 17 00:10:29.184700 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.184680    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj"
Sep 17 00:10:29.220645 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.220614    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474"
Sep 17 00:10:29.462886 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.462832    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69897745dd-nhpt7"]
Sep 17 00:10:29.466704 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:10:29.466679    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33131f95_e9e8_4d86_a9d8_73ff5abb5396.slice/crio-47278920e4e7d6993deb873f422cce635b633ebc29178dab370af32df800fc23 WatchSource:0}: Error finding container 47278920e4e7d6993deb873f422cce635b633ebc29178dab370af32df800fc23: Status 404 returned error can't find the container with id 47278920e4e7d6993deb873f422cce635b633ebc29178dab370af32df800fc23
Sep 17 00:10:29.490572 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.490535    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj"]
Sep 17 00:10:29.494294 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:10:29.494267    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55925826_a2f5_4c85_9fdd_66e886482069.slice/crio-c078eec4f69cce3a71011cfba7c92c85895504ad2e1ed7bb0dba453294b0a0b4 WatchSource:0}: Error finding container c078eec4f69cce3a71011cfba7c92c85895504ad2e1ed7bb0dba453294b0a0b4: Status 404 returned error can't find the container with id c078eec4f69cce3a71011cfba7c92c85895504ad2e1ed7bb0dba453294b0a0b4
Sep 17 00:10:29.530884 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.530860    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474"]
Sep 17 00:10:29.534645 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:10:29.534620    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84a8dad6_c057_4734_a357_421d90017acd.slice/crio-e010d6bb9c4c7e0d42f299fea54c6947d19f400d71170ba9167ccd791fc85804 WatchSource:0}: Error finding container e010d6bb9c4c7e0d42f299fea54c6947d19f400d71170ba9167ccd791fc85804: Status 404 returned error can't find the container with id e010d6bb9c4c7e0d42f299fea54c6947d19f400d71170ba9167ccd791fc85804
Sep 17 00:10:29.922477 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.922370    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474" event={"ID":"84a8dad6-c057-4734-a357-421d90017acd","Type":"ContainerStarted","Data":"e010d6bb9c4c7e0d42f299fea54c6947d19f400d71170ba9167ccd791fc85804"}
Sep 17 00:10:29.923575 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.923540    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj" event={"ID":"55925826-a2f5-4c85-9fdd-66e886482069","Type":"ContainerStarted","Data":"c078eec4f69cce3a71011cfba7c92c85895504ad2e1ed7bb0dba453294b0a0b4"}
Sep 17 00:10:29.926629 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.926606    2577 generic.go:356] "Generic (PLEG): container finished" podID="30b22991-ec04-4b85-9b9d-87959cfa30d1" containerID="2200cb2dceb13ae4b02ce9b98d050f79949060f50d64b6491131d01a7bed2303" exitCode=0
Sep 17 00:10:29.926736 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.926673    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4cmx" event={"ID":"30b22991-ec04-4b85-9b9d-87959cfa30d1","Type":"ContainerDied","Data":"2200cb2dceb13ae4b02ce9b98d050f79949060f50d64b6491131d01a7bed2303"}
Sep 17 00:10:29.927768 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:29.927736    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69897745dd-nhpt7" event={"ID":"33131f95-e9e8-4d86-a9d8-73ff5abb5396","Type":"ContainerStarted","Data":"47278920e4e7d6993deb873f422cce635b633ebc29178dab370af32df800fc23"}
Sep 17 00:10:30.346275 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:30.346234    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:30.346479 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:30.346301    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-cch7x\" (UID: \"b91229a2-6f57-4a69-af2c-8cd92b0459f0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x"
Sep 17 00:10:30.346479 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:30.346332    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls\") pod \"dns-default-zhxfj\" (UID: \"57f31c58-e354-4d56-8b7e-c55015c26176\") " pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:10:30.346604 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:30.346478    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 00:10:30.346604 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:30.346535    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 00:10:30.346604 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:30.346547    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls podName:57f31c58-e354-4d56-8b7e-c55015c26176 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:38.346528794 +0000 UTC m=+51.205021923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls") pod "dns-default-zhxfj" (UID: "57f31c58-e354-4d56-8b7e-c55015c26176") : secret "dns-default-metrics-tls" not found
Sep 17 00:10:30.346604 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:30.346550    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5957bbc947-plqh5: secret "image-registry-tls" not found
Sep 17 00:10:30.346604 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:30.346572    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 00:10:30.346851 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:30.346583    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls podName:5e823b26-89d1-4a1d-95b0-1d0002258005 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:38.346572964 +0000 UTC m=+51.205066091 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls") pod "image-registry-5957bbc947-plqh5" (UID: "5e823b26-89d1-4a1d-95b0-1d0002258005") : secret "image-registry-tls" not found
Sep 17 00:10:30.346851 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:30.346701    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert podName:b91229a2-6f57-4a69-af2c-8cd92b0459f0 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:38.346646514 +0000 UTC m=+51.205139627 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-cch7x" (UID: "b91229a2-6f57-4a69-af2c-8cd92b0459f0") : secret "networking-console-plugin-cert" not found
Sep 17 00:10:30.942465 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:30.942120    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4cmx" event={"ID":"30b22991-ec04-4b85-9b9d-87959cfa30d1","Type":"ContainerStarted","Data":"0b668916e56eda6ad2389b4b6ceb88e4716a541bea00f6ce7d72d2102886fa9b"}
Sep 17 00:10:30.964326 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:30.963618    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g4cmx" podStartSLOduration=8.003074281 podStartE2EDuration="42.963595754s" totalImagesPullingTime="34.960521473s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:09:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:10:30.961912396 +0000 UTC m=+43.820405544" watchObservedRunningTime="2026-09-17 00:10:30.963595754 +0000 UTC m=+43.822088892"
Sep 17 00:10:36.961841 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:36.961806    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69897745dd-nhpt7" event={"ID":"33131f95-e9e8-4d86-a9d8-73ff5abb5396","Type":"ContainerStarted","Data":"265b4933a00f21a14704e18f9efd5e0fa53b5ebd20cf53c2ff60a58da3573fe8"}
Sep 17 00:10:36.963931 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:36.963905    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474" event={"ID":"84a8dad6-c057-4734-a357-421d90017acd","Type":"ContainerStarted","Data":"f34b393fd0aff90f1dc4f278a6c12d27a736cd2e24726c7e0fd43dfffdb292de"}
Sep 17 00:10:36.964065 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:36.963943    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474" event={"ID":"84a8dad6-c057-4734-a357-421d90017acd","Type":"ContainerStarted","Data":"d66ddfea14154e32b573701354ca22ee902ad27313cc268e9aec0432fe2dfaf8"}
Sep 17 00:10:36.964065 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:36.963961    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474" event={"ID":"84a8dad6-c057-4734-a357-421d90017acd","Type":"ContainerStarted","Data":"c6d93b86a8ae05b080ec8ac413779849182736e32402e20c86931e142ae00502"}
Sep 17 00:10:36.965281 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:36.965258    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj" event={"ID":"55925826-a2f5-4c85-9fdd-66e886482069","Type":"ContainerStarted","Data":"33e9378bb4983c5de313835b8b691e22ebbd42556a8d9a5f9747d29a6b268ace"}
Sep 17 00:10:36.965471 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:36.965448    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj"
Sep 17 00:10:36.967079 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:36.967060    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj"
Sep 17 00:10:36.974984 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:36.974947    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69897745dd-nhpt7" podStartSLOduration=1.95906867 podStartE2EDuration="8.974935547s" totalImagesPullingTime="7.015866877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:10:36.974785342 +0000 UTC m=+49.833278476" watchObservedRunningTime="2026-09-17 00:10:36.974935547 +0000 UTC m=+49.833428682"
Sep 17 00:10:36.990530 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:36.990489    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj" podStartSLOduration=1.980729543 podStartE2EDuration="8.990476424s" totalImagesPullingTime="7.009746881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:10:36.989847699 +0000 UTC m=+49.848340845" watchObservedRunningTime="2026-09-17 00:10:36.990476424 +0000 UTC m=+49.848969558"
Sep 17 00:10:37.007408 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:37.007369    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474" podStartSLOduration=2.036847268 podStartE2EDuration="9.007356865s" totalImagesPullingTime="6.970509597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:10:37.00676814 +0000 UTC m=+49.865261276" watchObservedRunningTime="2026-09-17 00:10:37.007356865 +0000 UTC m=+49.865849999"
Sep 17 00:10:38.410993 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:38.410946    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:38.411400 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:38.411007    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-cch7x\" (UID: \"b91229a2-6f57-4a69-af2c-8cd92b0459f0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x"
Sep 17 00:10:38.411400 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:38.411037    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls\") pod \"dns-default-zhxfj\" (UID: \"57f31c58-e354-4d56-8b7e-c55015c26176\") " pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:10:38.411400 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:38.411103    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 00:10:38.411400 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:38.411126    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5957bbc947-plqh5: secret "image-registry-tls" not found
Sep 17 00:10:38.411400 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:38.411177    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls podName:5e823b26-89d1-4a1d-95b0-1d0002258005 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:54.411162029 +0000 UTC m=+67.269655164 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls") pod "image-registry-5957bbc947-plqh5" (UID: "5e823b26-89d1-4a1d-95b0-1d0002258005") : secret "image-registry-tls" not found
Sep 17 00:10:38.411400 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:38.411180    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 00:10:38.411400 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:38.411100    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 00:10:38.411400 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:38.411245    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls podName:57f31c58-e354-4d56-8b7e-c55015c26176 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:54.41122649 +0000 UTC m=+67.269719604 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls") pod "dns-default-zhxfj" (UID: "57f31c58-e354-4d56-8b7e-c55015c26176") : secret "dns-default-metrics-tls" not found
Sep 17 00:10:38.411400 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:38.411259    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert podName:b91229a2-6f57-4a69-af2c-8cd92b0459f0 nodeName:}" failed. No retries permitted until 2026-09-17 00:10:54.411252617 +0000 UTC m=+67.269745729 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-cch7x" (UID: "b91229a2-6f57-4a69-af2c-8cd92b0459f0") : secret "networking-console-plugin-cert" not found
Sep 17 00:10:47.895619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:47.895589    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bz685"
Sep 17 00:10:52.413176 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:52.413127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs5sk\" (UniqueName: \"kubernetes.io/projected/f5f1a947-783b-4a35-a70b-b850883fef06-kube-api-access-gs5sk\") pod \"network-check-target-s8lnt\" (UID: \"f5f1a947-783b-4a35-a70b-b850883fef06\") " pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:10:52.413176 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:52.413190    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/504ae0de-dca4-48aa-964c-b457c02267a5-original-pull-secret\") pod \"global-pull-secret-syncer-ch5w2\" (UID: \"504ae0de-dca4-48aa-964c-b457c02267a5\") " pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:10:52.413774 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:52.413356    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs\") pod \"network-metrics-daemon-xbhz5\" (UID: \"b956a62e-c4ac-4e1d-aeb0-3f7907a0a702\") " pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:10:52.416243 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:52.416217    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 00:10:52.416348 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:52.416332    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 00:10:52.416392 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:52.416356    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 00:10:52.423578 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:52.423552    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 00:10:52.423675 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:52.423606    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs podName:b956a62e-c4ac-4e1d-aeb0-3f7907a0a702 nodeName:}" failed. No retries permitted until 2026-09-17 00:11:56.423589562 +0000 UTC m=+129.282082681 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs") pod "network-metrics-daemon-xbhz5" (UID: "b956a62e-c4ac-4e1d-aeb0-3f7907a0a702") : secret "metrics-daemon-secret" not found
Sep 17 00:10:52.425455 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:52.425438    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 00:10:52.425514 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:52.425480    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/504ae0de-dca4-48aa-964c-b457c02267a5-original-pull-secret\") pod \"global-pull-secret-syncer-ch5w2\" (UID: \"504ae0de-dca4-48aa-964c-b457c02267a5\") " pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:10:52.436385 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:52.436362    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs5sk\" (UniqueName: \"kubernetes.io/projected/f5f1a947-783b-4a35-a70b-b850883fef06-kube-api-access-gs5sk\") pod \"network-check-target-s8lnt\" (UID: \"f5f1a947-783b-4a35-a70b-b850883fef06\") " pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:10:52.717484 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:52.717456    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-86j4t\""
Sep 17 00:10:52.725569 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:52.725546    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:10:52.725569 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:52.725555    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch5w2"
Sep 17 00:10:53.003716 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:53.003636    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s8lnt"]
Sep 17 00:10:53.004368 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:53.004350    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ch5w2"]
Sep 17 00:10:53.006899 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:10:53.006865    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504ae0de_dca4_48aa_964c_b457c02267a5.slice/crio-76986257883fcc4de0d409bbb69f5b4d36b94c6e6e26516728b8760997dc5ae1 WatchSource:0}: Error finding container 76986257883fcc4de0d409bbb69f5b4d36b94c6e6e26516728b8760997dc5ae1: Status 404 returned error can't find the container with id 76986257883fcc4de0d409bbb69f5b4d36b94c6e6e26516728b8760997dc5ae1
Sep 17 00:10:53.007539 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:10:53.007517    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f1a947_783b_4a35_a70b_b850883fef06.slice/crio-47dd6f4317a5fe3edad9c14a9e715414fd762ea0ccc18a4de224c048acda4161 WatchSource:0}: Error finding container 47dd6f4317a5fe3edad9c14a9e715414fd762ea0ccc18a4de224c048acda4161: Status 404 returned error can't find the container with id 47dd6f4317a5fe3edad9c14a9e715414fd762ea0ccc18a4de224c048acda4161
Sep 17 00:10:53.017932 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:53.017904    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s8lnt" event={"ID":"f5f1a947-783b-4a35-a70b-b850883fef06","Type":"ContainerStarted","Data":"47dd6f4317a5fe3edad9c14a9e715414fd762ea0ccc18a4de224c048acda4161"}
Sep 17 00:10:53.019023 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:53.018995    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ch5w2" event={"ID":"504ae0de-dca4-48aa-964c-b457c02267a5","Type":"ContainerStarted","Data":"76986257883fcc4de0d409bbb69f5b4d36b94c6e6e26516728b8760997dc5ae1"}
Sep 17 00:10:54.429812 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:54.429753    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:10:54.430260 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:54.429823    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-cch7x\" (UID: \"b91229a2-6f57-4a69-af2c-8cd92b0459f0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x"
Sep 17 00:10:54.430260 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:54.429855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls\") pod \"dns-default-zhxfj\" (UID: \"57f31c58-e354-4d56-8b7e-c55015c26176\") " pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:10:54.430260 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:54.429933    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 00:10:54.430260 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:54.429957    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5957bbc947-plqh5: secret "image-registry-tls" not found
Sep 17 00:10:54.430260 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:54.429967    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 00:10:54.430260 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:54.430033    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls podName:57f31c58-e354-4d56-8b7e-c55015c26176 nodeName:}" failed. No retries permitted until 2026-09-17 00:11:26.430014633 +0000 UTC m=+99.288507768 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls") pod "dns-default-zhxfj" (UID: "57f31c58-e354-4d56-8b7e-c55015c26176") : secret "dns-default-metrics-tls" not found
Sep 17 00:10:54.430260 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:54.430051    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls podName:5e823b26-89d1-4a1d-95b0-1d0002258005 nodeName:}" failed. No retries permitted until 2026-09-17 00:11:26.430041108 +0000 UTC m=+99.288534229 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls") pod "image-registry-5957bbc947-plqh5" (UID: "5e823b26-89d1-4a1d-95b0-1d0002258005") : secret "image-registry-tls" not found
Sep 17 00:10:54.430260 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:54.430089    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 00:10:54.430260 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:10:54.430130    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert podName:b91229a2-6f57-4a69-af2c-8cd92b0459f0 nodeName:}" failed. No retries permitted until 2026-09-17 00:11:26.430112053 +0000 UTC m=+99.288605170 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-cch7x" (UID: "b91229a2-6f57-4a69-af2c-8cd92b0459f0") : secret "networking-console-plugin-cert" not found
Sep 17 00:10:58.036061 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:58.035970    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ch5w2" event={"ID":"504ae0de-dca4-48aa-964c-b457c02267a5","Type":"ContainerStarted","Data":"b2a00a71626215a5e5cad3458f3caa9b720d7621a05731c7cf89207d85cddca9"}
Sep 17 00:10:58.037773 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:58.037742    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s8lnt" event={"ID":"f5f1a947-783b-4a35-a70b-b850883fef06","Type":"ContainerStarted","Data":"59685bdadd9fc52795c1600beb6cc42a4cb95d4ddd564d4b5376eb300088219d"}
Sep 17 00:10:58.037903 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:58.037802    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:10:58.049809 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:58.049762    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ch5w2" podStartSLOduration=65.266094251 podStartE2EDuration="1m10.049747286s" totalImagesPullingTime="4.783653035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:10:58.048441878 +0000 UTC m=+70.906935009" watchObservedRunningTime="2026-09-17 00:10:58.049747286 +0000 UTC m=+70.908240454"
Sep 17 00:10:58.064247 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:10:58.064106    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-s8lnt" podStartSLOduration=65.286087245 podStartE2EDuration="1m10.064091287s" totalImagesPullingTime="4.778004042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:10:58.063985227 +0000 UTC m=+70.922478362" watchObservedRunningTime="2026-09-17 00:10:58.064091287 +0000 UTC m=+70.922584422"
Sep 17 00:11:26.466853 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:11:26.466801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-cch7x\" (UID: \"b91229a2-6f57-4a69-af2c-8cd92b0459f0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x"
Sep 17 00:11:26.466853 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:11:26.466853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls\") pod \"dns-default-zhxfj\" (UID: \"57f31c58-e354-4d56-8b7e-c55015c26176\") " pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:11:26.467472 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:11:26.466903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:11:26.467472 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:11:26.466970    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 00:11:26.467472 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:11:26.467025    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 00:11:26.467472 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:11:26.467057    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert podName:b91229a2-6f57-4a69-af2c-8cd92b0459f0 nodeName:}" failed. No retries permitted until 2026-09-17 00:12:30.467036382 +0000 UTC m=+163.325529519 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-cch7x" (UID: "b91229a2-6f57-4a69-af2c-8cd92b0459f0") : secret "networking-console-plugin-cert" not found
Sep 17 00:11:26.467472 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:11:26.466981    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 00:11:26.467472 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:11:26.467076    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5957bbc947-plqh5: secret "image-registry-tls" not found
Sep 17 00:11:26.467472 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:11:26.467094    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls podName:57f31c58-e354-4d56-8b7e-c55015c26176 nodeName:}" failed. No retries permitted until 2026-09-17 00:12:30.467080082 +0000 UTC m=+163.325573194 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls") pod "dns-default-zhxfj" (UID: "57f31c58-e354-4d56-8b7e-c55015c26176") : secret "dns-default-metrics-tls" not found
Sep 17 00:11:26.467472 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:11:26.467119    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls podName:5e823b26-89d1-4a1d-95b0-1d0002258005 nodeName:}" failed. No retries permitted until 2026-09-17 00:12:30.46710944 +0000 UTC m=+163.325602578 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls") pod "image-registry-5957bbc947-plqh5" (UID: "5e823b26-89d1-4a1d-95b0-1d0002258005") : secret "image-registry-tls" not found
Sep 17 00:11:29.043530 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:11:29.043492    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-s8lnt"
Sep 17 00:11:56.497844 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:11:56.497802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs\") pod \"network-metrics-daemon-xbhz5\" (UID: \"b956a62e-c4ac-4e1d-aeb0-3f7907a0a702\") " pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:11:56.498326 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:11:56.497954    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 00:11:56.498326 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:11:56.498021    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs podName:b956a62e-c4ac-4e1d-aeb0-3f7907a0a702 nodeName:}" failed. No retries permitted until 2026-09-17 00:13:58.498004433 +0000 UTC m=+251.356497559 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs") pod "network-metrics-daemon-xbhz5" (UID: "b956a62e-c4ac-4e1d-aeb0-3f7907a0a702") : secret "metrics-daemon-secret" not found
Sep 17 00:12:25.557252 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:12:25.557161    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x" podUID="b91229a2-6f57-4a69-af2c-8cd92b0459f0"
Sep 17 00:12:25.574353 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:12:25.574326    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5957bbc947-plqh5" podUID="5e823b26-89d1-4a1d-95b0-1d0002258005"
Sep 17 00:12:25.613822 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:12:25.613786    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zhxfj" podUID="57f31c58-e354-4d56-8b7e-c55015c26176"
Sep 17 00:12:26.293324 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:12:26.293291    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:12:26.293505 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:12:26.293291    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:12:26.293505 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:12:26.293291    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x"
Sep 17 00:12:26.620347 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:12:26.620272    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-xbhz5" podUID="b956a62e-c4ac-4e1d-aeb0-3f7907a0a702"
Sep 17 00:12:30.544115 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:12:30.544068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:12:30.544115 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:12:30.544128    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-cch7x\" (UID: \"b91229a2-6f57-4a69-af2c-8cd92b0459f0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x"
Sep 17 00:12:30.544581 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:12:30.544155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls\") pod \"dns-default-zhxfj\" (UID: \"57f31c58-e354-4d56-8b7e-c55015c26176\") " pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:12:30.544581 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:12:30.544226    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 00:12:30.544581 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:12:30.544239    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 00:12:30.544581 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:12:30.544266    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 00:12:30.544581 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:12:30.544245    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5957bbc947-plqh5: secret "image-registry-tls" not found
Sep 17 00:12:30.544581 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:12:30.544315    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert podName:b91229a2-6f57-4a69-af2c-8cd92b0459f0 nodeName:}" failed. No retries permitted until 2026-09-17 00:14:32.544295838 +0000 UTC m=+285.402788964 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-cch7x" (UID: "b91229a2-6f57-4a69-af2c-8cd92b0459f0") : secret "networking-console-plugin-cert" not found
Sep 17 00:12:30.544581 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:12:30.544333    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls podName:57f31c58-e354-4d56-8b7e-c55015c26176 nodeName:}" failed. No retries permitted until 2026-09-17 00:14:32.544324658 +0000 UTC m=+285.402817776 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls") pod "dns-default-zhxfj" (UID: "57f31c58-e354-4d56-8b7e-c55015c26176") : secret "dns-default-metrics-tls" not found
Sep 17 00:12:30.544581 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:12:30.544347    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls podName:5e823b26-89d1-4a1d-95b0-1d0002258005 nodeName:}" failed. No retries permitted until 2026-09-17 00:14:32.544339432 +0000 UTC m=+285.402832550 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls") pod "image-registry-5957bbc947-plqh5" (UID: "5e823b26-89d1-4a1d-95b0-1d0002258005") : secret "image-registry-tls" not found
Sep 17 00:12:36.966694 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:12:36.966655    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj" podUID="55925826-a2f5-4c85-9fdd-66e886482069" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.11:8000/readyz\": dial tcp 10.132.0.11:8000: connect: connection refused"
Sep 17 00:12:37.326918 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:12:37.326835    2577 generic.go:356] "Generic (PLEG): container finished" podID="55925826-a2f5-4c85-9fdd-66e886482069" containerID="33e9378bb4983c5de313835b8b691e22ebbd42556a8d9a5f9747d29a6b268ace" exitCode=1
Sep 17 00:12:37.327058 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:12:37.326912    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj" event={"ID":"55925826-a2f5-4c85-9fdd-66e886482069","Type":"ContainerDied","Data":"33e9378bb4983c5de313835b8b691e22ebbd42556a8d9a5f9747d29a6b268ace"}
Sep 17 00:12:37.327289 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:12:37.327267    2577 scope.go:117] "RemoveContainer" containerID="33e9378bb4983c5de313835b8b691e22ebbd42556a8d9a5f9747d29a6b268ace"
Sep 17 00:12:37.328366 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:12:37.328340    2577 generic.go:356] "Generic (PLEG): container finished" podID="33131f95-e9e8-4d86-a9d8-73ff5abb5396" containerID="265b4933a00f21a14704e18f9efd5e0fa53b5ebd20cf53c2ff60a58da3573fe8" exitCode=255
Sep 17 00:12:37.328475 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:12:37.328373    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69897745dd-nhpt7" event={"ID":"33131f95-e9e8-4d86-a9d8-73ff5abb5396","Type":"ContainerDied","Data":"265b4933a00f21a14704e18f9efd5e0fa53b5ebd20cf53c2ff60a58da3573fe8"}
Sep 17 00:12:37.328635 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:12:37.328622    2577 scope.go:117] "RemoveContainer" containerID="265b4933a00f21a14704e18f9efd5e0fa53b5ebd20cf53c2ff60a58da3573fe8"
Sep 17 00:12:38.333289 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:12:38.333255    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69897745dd-nhpt7" event={"ID":"33131f95-e9e8-4d86-a9d8-73ff5abb5396","Type":"ContainerStarted","Data":"0c7b359582245fb34f4f272619363cd9e5c1d7b50771b4bca4059f578cc319c7"}
Sep 17 00:12:38.334849 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:12:38.334824    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj" event={"ID":"55925826-a2f5-4c85-9fdd-66e886482069","Type":"ContainerStarted","Data":"49ee82279ea218f412a758a5fe27deb18e3f6a7b3342ee834aa0dc495673c0ef"}
Sep 17 00:12:38.335111 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:12:38.335085    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj"
Sep 17 00:12:38.335617 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:12:38.335599    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj"
Sep 17 00:12:38.601963 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:12:38.601865    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:13:56.099301 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:56.099267    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7rtx9"]
Sep 17 00:13:56.101258 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:56.101239    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:13:56.103696 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:56.103669    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 00:13:56.103841 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:56.103675    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 00:13:56.103841 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:56.103675    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 00:13:56.104087 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:56.104072    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 00:13:56.104954 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:56.104938    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2jxk2\""
Sep 17 00:13:56.113271 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:56.113249    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7rtx9"]
Sep 17 00:13:56.163018 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:56.162972    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d564ffa-7392-4e02-b8fb-34c857ae3874-crio-socket\") pod \"insights-runtime-extractor-7rtx9\" (UID: \"8d564ffa-7392-4e02-b8fb-34c857ae3874\") " pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:13:56.163018 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:56.163019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d564ffa-7392-4e02-b8fb-34c857ae3874-data-volume\") pod \"insights-runtime-extractor-7rtx9\" (UID: \"8d564ffa-7392-4e02-b8fb-34c857ae3874\") " pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:13:56.163225 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:56.163075    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d564ffa-7392-4e02-b8fb-34c857ae3874-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7rtx9\" (UID: \"8d564ffa-7392-4e02-b8fb-34c857ae3874\") " pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:13:56.163225 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:56.163102    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqvb6\" (UniqueName: \"kubernetes.io/projected/8d564ffa-7392-4e02-b8fb-34c857ae3874-kube-api-access-gqvb6\") pod \"insights-runtime-extractor-7rtx9\" (UID: \"8d564ffa-7392-4e02-b8fb-34c857ae3874\") " pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:13:56.163225 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:56.163128    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d564ffa-7392-4e02-b8fb-34c857ae3874-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rtx9\" (UID: \"8d564ffa-7392-4e02-b8fb-34c857ae3874\") " pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:13:56.264344 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:56.264291    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d564ffa-7392-4e02-b8fb-34c857ae3874-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7rtx9\" (UID: \"8d564ffa-7392-4e02-b8fb-34c857ae3874\") " pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:13:56.264344 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:56.264345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqvb6\" (UniqueName: \"kubernetes.io/projected/8d564ffa-7392-4e02-b8fb-34c857ae3874-kube-api-access-gqvb6\") pod \"insights-runtime-extractor-7rtx9\" (UID: \"8d564ffa-7392-4e02-b8fb-34c857ae3874\") " pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:13:56.264572 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:56.264370    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d564ffa-7392-4e02-b8fb-34c857ae3874-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rtx9\" (UID: \"8d564ffa-7392-4e02-b8fb-34c857ae3874\") " pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:13:56.264572 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:13:56.264437    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 00:13:56.264572 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:13:56.264475    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 00:13:56.264572 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:13:56.264501    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d564ffa-7392-4e02-b8fb-34c857ae3874-kube-rbac-proxy-cm podName:8d564ffa-7392-4e02-b8fb-34c857ae3874 nodeName:}" failed. No retries permitted until 2026-09-17 00:13:56.764487588 +0000 UTC m=+249.622980701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8d564ffa-7392-4e02-b8fb-34c857ae3874-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7rtx9" (UID: "8d564ffa-7392-4e02-b8fb-34c857ae3874") : configmap "kube-rbac-proxy" not found
Sep 17 00:13:56.264572 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:13:56.264523    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d564ffa-7392-4e02-b8fb-34c857ae3874-insights-runtime-extractor-tls podName:8d564ffa-7392-4e02-b8fb-34c857ae3874 nodeName:}" failed. No retries permitted until 2026-09-17 00:13:56.764511609 +0000 UTC m=+249.623004722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d564ffa-7392-4e02-b8fb-34c857ae3874-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7rtx9" (UID: "8d564ffa-7392-4e02-b8fb-34c857ae3874") : secret "insights-runtime-extractor-tls" not found
Sep 17 00:13:56.264572 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:56.264555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d564ffa-7392-4e02-b8fb-34c857ae3874-crio-socket\") pod \"insights-runtime-extractor-7rtx9\" (UID: \"8d564ffa-7392-4e02-b8fb-34c857ae3874\") " pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:13:56.264572 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:56.264575    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d564ffa-7392-4e02-b8fb-34c857ae3874-data-volume\") pod \"insights-runtime-extractor-7rtx9\" (UID: \"8d564ffa-7392-4e02-b8fb-34c857ae3874\") " pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:13:56.264878 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:56.264648    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d564ffa-7392-4e02-b8fb-34c857ae3874-crio-socket\") pod \"insights-runtime-extractor-7rtx9\" (UID: \"8d564ffa-7392-4e02-b8fb-34c857ae3874\") " pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:13:56.264878 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:56.264855    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d564ffa-7392-4e02-b8fb-34c857ae3874-data-volume\") pod \"insights-runtime-extractor-7rtx9\" (UID: \"8d564ffa-7392-4e02-b8fb-34c857ae3874\") " pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:13:56.272445 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:56.272401    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqvb6\" (UniqueName: \"kubernetes.io/projected/8d564ffa-7392-4e02-b8fb-34c857ae3874-kube-api-access-gqvb6\") pod \"insights-runtime-extractor-7rtx9\" (UID: \"8d564ffa-7392-4e02-b8fb-34c857ae3874\") " pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:13:56.769046 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:56.768984    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d564ffa-7392-4e02-b8fb-34c857ae3874-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rtx9\" (UID: \"8d564ffa-7392-4e02-b8fb-34c857ae3874\") " pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:13:56.769238 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:56.769099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d564ffa-7392-4e02-b8fb-34c857ae3874-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7rtx9\" (UID: \"8d564ffa-7392-4e02-b8fb-34c857ae3874\") " pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:13:56.769238 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:13:56.769143    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 00:13:56.769238 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:13:56.769162    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 00:13:56.769238 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:13:56.769211    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d564ffa-7392-4e02-b8fb-34c857ae3874-insights-runtime-extractor-tls podName:8d564ffa-7392-4e02-b8fb-34c857ae3874 nodeName:}" failed. No retries permitted until 2026-09-17 00:13:57.769195382 +0000 UTC m=+250.627688495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d564ffa-7392-4e02-b8fb-34c857ae3874-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7rtx9" (UID: "8d564ffa-7392-4e02-b8fb-34c857ae3874") : secret "insights-runtime-extractor-tls" not found
Sep 17 00:13:56.769238 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:13:56.769225    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d564ffa-7392-4e02-b8fb-34c857ae3874-kube-rbac-proxy-cm podName:8d564ffa-7392-4e02-b8fb-34c857ae3874 nodeName:}" failed. No retries permitted until 2026-09-17 00:13:57.769219883 +0000 UTC m=+250.627712997 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8d564ffa-7392-4e02-b8fb-34c857ae3874-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7rtx9" (UID: "8d564ffa-7392-4e02-b8fb-34c857ae3874") : configmap "kube-rbac-proxy" not found
Sep 17 00:13:57.777303 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:57.777266    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d564ffa-7392-4e02-b8fb-34c857ae3874-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7rtx9\" (UID: \"8d564ffa-7392-4e02-b8fb-34c857ae3874\") " pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:13:57.777760 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:57.777328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d564ffa-7392-4e02-b8fb-34c857ae3874-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rtx9\" (UID: \"8d564ffa-7392-4e02-b8fb-34c857ae3874\") " pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:13:57.777760 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:13:57.777395    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 00:13:57.777760 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:13:57.777491    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d564ffa-7392-4e02-b8fb-34c857ae3874-kube-rbac-proxy-cm podName:8d564ffa-7392-4e02-b8fb-34c857ae3874 nodeName:}" failed. No retries permitted until 2026-09-17 00:13:59.777470029 +0000 UTC m=+252.635963146 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8d564ffa-7392-4e02-b8fb-34c857ae3874-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7rtx9" (UID: "8d564ffa-7392-4e02-b8fb-34c857ae3874") : configmap "kube-rbac-proxy" not found
Sep 17 00:13:57.777760 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:13:57.777505    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 00:13:57.777760 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:13:57.777543    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d564ffa-7392-4e02-b8fb-34c857ae3874-insights-runtime-extractor-tls podName:8d564ffa-7392-4e02-b8fb-34c857ae3874 nodeName:}" failed. No retries permitted until 2026-09-17 00:13:59.777532214 +0000 UTC m=+252.636025344 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d564ffa-7392-4e02-b8fb-34c857ae3874-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7rtx9" (UID: "8d564ffa-7392-4e02-b8fb-34c857ae3874") : secret "insights-runtime-extractor-tls" not found
Sep 17 00:13:58.083966 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:58.083890    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9xxfq_3e97bfc2-d46a-4384-bc1a-a1a2d37afe75/dns-node-resolver/0.log"
Sep 17 00:13:58.584779 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:58.584738    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs\") pod \"network-metrics-daemon-xbhz5\" (UID: \"b956a62e-c4ac-4e1d-aeb0-3f7907a0a702\") " pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:13:58.584986 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:13:58.584909    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 00:13:58.585053 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:13:58.584986    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs podName:b956a62e-c4ac-4e1d-aeb0-3f7907a0a702 nodeName:}" failed. No retries permitted until 2026-09-17 00:16:00.584965196 +0000 UTC m=+373.443458308 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs") pod "network-metrics-daemon-xbhz5" (UID: "b956a62e-c4ac-4e1d-aeb0-3f7907a0a702") : secret "metrics-daemon-secret" not found
Sep 17 00:13:59.083912 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:59.083881    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rg8zr_e6570468-fe3a-4512-8bda-9880edc3ff96/node-ca/0.log"
Sep 17 00:13:59.796779 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:59.796734    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d564ffa-7392-4e02-b8fb-34c857ae3874-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rtx9\" (UID: \"8d564ffa-7392-4e02-b8fb-34c857ae3874\") " pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:13:59.796961 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:59.796836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d564ffa-7392-4e02-b8fb-34c857ae3874-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7rtx9\" (UID: \"8d564ffa-7392-4e02-b8fb-34c857ae3874\") " pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:13:59.796961 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:13:59.796885    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 00:13:59.796961 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:13:59.796953    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d564ffa-7392-4e02-b8fb-34c857ae3874-insights-runtime-extractor-tls podName:8d564ffa-7392-4e02-b8fb-34c857ae3874 nodeName:}" failed. No retries permitted until 2026-09-17 00:14:03.79693742 +0000 UTC m=+256.655430534 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d564ffa-7392-4e02-b8fb-34c857ae3874-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7rtx9" (UID: "8d564ffa-7392-4e02-b8fb-34c857ae3874") : secret "insights-runtime-extractor-tls" not found
Sep 17 00:13:59.797297 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:13:59.797279    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d564ffa-7392-4e02-b8fb-34c857ae3874-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7rtx9\" (UID: \"8d564ffa-7392-4e02-b8fb-34c857ae3874\") " pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:14:00.083795 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:00.083721    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-twmbk_ccc9a7ce-9b6a-40f4-9d5e-388c3e892f11/serve-healthcheck-canary/0.log"
Sep 17 00:14:03.830761 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:03.830725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d564ffa-7392-4e02-b8fb-34c857ae3874-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rtx9\" (UID: \"8d564ffa-7392-4e02-b8fb-34c857ae3874\") " pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:14:03.831237 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:14:03.830865    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 00:14:03.831237 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:14:03.830926    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d564ffa-7392-4e02-b8fb-34c857ae3874-insights-runtime-extractor-tls podName:8d564ffa-7392-4e02-b8fb-34c857ae3874 nodeName:}" failed. No retries permitted until 2026-09-17 00:14:11.830908725 +0000 UTC m=+264.689401856 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d564ffa-7392-4e02-b8fb-34c857ae3874-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7rtx9" (UID: "8d564ffa-7392-4e02-b8fb-34c857ae3874") : secret "insights-runtime-extractor-tls" not found
Sep 17 00:14:11.898899 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:11.898816    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d564ffa-7392-4e02-b8fb-34c857ae3874-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rtx9\" (UID: \"8d564ffa-7392-4e02-b8fb-34c857ae3874\") " pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:14:11.901076 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:11.901053    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d564ffa-7392-4e02-b8fb-34c857ae3874-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7rtx9\" (UID: \"8d564ffa-7392-4e02-b8fb-34c857ae3874\") " pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:14:12.010606 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:12.010573    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7rtx9"
Sep 17 00:14:12.276341 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:12.276293    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7rtx9"]
Sep 17 00:14:12.280648 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:14:12.280623    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d564ffa_7392_4e02_b8fb_34c857ae3874.slice/crio-f01f00fc548ac8e88736116ca6997e7c3c415328e1fb764a1d954a696bb6ec50 WatchSource:0}: Error finding container f01f00fc548ac8e88736116ca6997e7c3c415328e1fb764a1d954a696bb6ec50: Status 404 returned error can't find the container with id f01f00fc548ac8e88736116ca6997e7c3c415328e1fb764a1d954a696bb6ec50
Sep 17 00:14:12.608617 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:12.608525    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7rtx9" event={"ID":"8d564ffa-7392-4e02-b8fb-34c857ae3874","Type":"ContainerStarted","Data":"4488b6dfdeab1a941b35eef377876338fa5b4003f4ba7f753df4343a6be1be72"}
Sep 17 00:14:12.608617 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:12.608581    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7rtx9" event={"ID":"8d564ffa-7392-4e02-b8fb-34c857ae3874","Type":"ContainerStarted","Data":"f01f00fc548ac8e88736116ca6997e7c3c415328e1fb764a1d954a696bb6ec50"}
Sep 17 00:14:13.612666 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:13.612629    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7rtx9" event={"ID":"8d564ffa-7392-4e02-b8fb-34c857ae3874","Type":"ContainerStarted","Data":"171bb017991c75d6dc4eb3f2d740e6621d1130382db75f85153194452a633b58"}
Sep 17 00:14:15.620745 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:15.620701    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7rtx9" event={"ID":"8d564ffa-7392-4e02-b8fb-34c857ae3874","Type":"ContainerStarted","Data":"3d24514ebb7bdaf4f2cbe01dd6146984b41bc60b2be98eab0d45a8169332ee2e"}
Sep 17 00:14:15.640141 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:15.640095    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7rtx9" podStartSLOduration=17.30634718 podStartE2EDuration="19.640080339s" totalImagesPullingTime="2.333733159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:13:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:14:15.639518836 +0000 UTC m=+268.498011972" watchObservedRunningTime="2026-09-17 00:14:15.640080339 +0000 UTC m=+268.498573474"
Sep 17 00:14:29.294728 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:14:29.294675    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x" podUID="b91229a2-6f57-4a69-af2c-8cd92b0459f0"
Sep 17 00:14:29.294728 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:14:29.294675    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5957bbc947-plqh5" podUID="5e823b26-89d1-4a1d-95b0-1d0002258005"
Sep 17 00:14:29.294728 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:14:29.294673    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zhxfj" podUID="57f31c58-e354-4d56-8b7e-c55015c26176"
Sep 17 00:14:29.660742 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:29.660657    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x"
Sep 17 00:14:29.660742 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:29.660684    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:14:29.660939 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:29.660807    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:14:32.556717 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:32.556686    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-cch7x\" (UID: \"b91229a2-6f57-4a69-af2c-8cd92b0459f0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x"
Sep 17 00:14:32.557181 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:32.556724    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:14:32.557181 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:32.556768    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls\") pod \"dns-default-zhxfj\" (UID: \"57f31c58-e354-4d56-8b7e-c55015c26176\") " pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:14:32.559200 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:32.559176    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57f31c58-e354-4d56-8b7e-c55015c26176-metrics-tls\") pod \"dns-default-zhxfj\" (UID: \"57f31c58-e354-4d56-8b7e-c55015c26176\") " pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:14:32.559264 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:32.559185    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b91229a2-6f57-4a69-af2c-8cd92b0459f0-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-cch7x\" (UID: \"b91229a2-6f57-4a69-af2c-8cd92b0459f0\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x"
Sep 17 00:14:32.559341 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:32.559326    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls\") pod \"image-registry-5957bbc947-plqh5\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") " pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:14:32.663713 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:32.663678    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j25ch\""
Sep 17 00:14:32.663713 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:32.663678    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-8njzj\""
Sep 17 00:14:32.663910 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:32.663693    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d74lr\""
Sep 17 00:14:32.671769 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:32.671747    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x"
Sep 17 00:14:32.671889 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:32.671840    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:14:32.671889 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:32.671883    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:14:32.958982 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:32.958953    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5957bbc947-plqh5"]
Sep 17 00:14:32.961842 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:32.961816    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x"]
Sep 17 00:14:32.964280 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:14:32.964248    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e823b26_89d1_4a1d_95b0_1d0002258005.slice/crio-7bbec06d0bc7b189160710d3378bb054012d931c8d167c3637753781e9cbb040 WatchSource:0}: Error finding container 7bbec06d0bc7b189160710d3378bb054012d931c8d167c3637753781e9cbb040: Status 404 returned error can't find the container with id 7bbec06d0bc7b189160710d3378bb054012d931c8d167c3637753781e9cbb040
Sep 17 00:14:32.964974 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:32.964939    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zhxfj"]
Sep 17 00:14:32.965136 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:14:32.965106    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91229a2_6f57_4a69_af2c_8cd92b0459f0.slice/crio-3daf816ae5cc3ee0ed2cb3b63e7df4f854a8c70774faf46b9ec3c624818f70b1 WatchSource:0}: Error finding container 3daf816ae5cc3ee0ed2cb3b63e7df4f854a8c70774faf46b9ec3c624818f70b1: Status 404 returned error can't find the container with id 3daf816ae5cc3ee0ed2cb3b63e7df4f854a8c70774faf46b9ec3c624818f70b1
Sep 17 00:14:32.968144 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:14:32.968122    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f31c58_e354_4d56_8b7e_c55015c26176.slice/crio-f2c3abf454140c8d8e2419df2c1becd21a57aad3b9108a20056b6e21e8f03cc1 WatchSource:0}: Error finding container f2c3abf454140c8d8e2419df2c1becd21a57aad3b9108a20056b6e21e8f03cc1: Status 404 returned error can't find the container with id f2c3abf454140c8d8e2419df2c1becd21a57aad3b9108a20056b6e21e8f03cc1
Sep 17 00:14:33.674372 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:33.674275    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5957bbc947-plqh5" event={"ID":"5e823b26-89d1-4a1d-95b0-1d0002258005","Type":"ContainerStarted","Data":"a0bff7f87e60e1e5d5a4d28a6d8b43bab797b30aa3eef2cf053870a53ab1965b"}
Sep 17 00:14:33.674372 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:33.674343    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5957bbc947-plqh5" event={"ID":"5e823b26-89d1-4a1d-95b0-1d0002258005","Type":"ContainerStarted","Data":"7bbec06d0bc7b189160710d3378bb054012d931c8d167c3637753781e9cbb040"}
Sep 17 00:14:33.674925 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:33.674387    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:14:33.675667 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:33.675642    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zhxfj" event={"ID":"57f31c58-e354-4d56-8b7e-c55015c26176","Type":"ContainerStarted","Data":"f2c3abf454140c8d8e2419df2c1becd21a57aad3b9108a20056b6e21e8f03cc1"}
Sep 17 00:14:33.676899 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:33.676875    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x" event={"ID":"b91229a2-6f57-4a69-af2c-8cd92b0459f0","Type":"ContainerStarted","Data":"3daf816ae5cc3ee0ed2cb3b63e7df4f854a8c70774faf46b9ec3c624818f70b1"}
Sep 17 00:14:33.694532 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:33.694477    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5957bbc947-plqh5" podStartSLOduration=285.694460868 podStartE2EDuration="4m45.694460868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:14:33.693328228 +0000 UTC m=+286.551821358" watchObservedRunningTime="2026-09-17 00:14:33.694460868 +0000 UTC m=+286.552954000"
Sep 17 00:14:35.303654 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.303622    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xj545"]
Sep 17 00:14:35.305756 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.305739    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.308560 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.308533    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 00:14:35.308692 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.308537    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 00:14:35.309666 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.309648    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 00:14:35.309792 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.309680    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 00:14:35.309792 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.309729    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 00:14:35.309792 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.309745    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 00:14:35.309906 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.309797    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-klwx5\""
Sep 17 00:14:35.379081 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.379050    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-node-exporter-accelerators-collector-config\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.379081 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.379085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-node-exporter-textfile\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.379284 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.379107    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-node-exporter-wtmp\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.379284 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.379124    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf6vc\" (UniqueName: \"kubernetes.io/projected/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-kube-api-access-lf6vc\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.379284 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.379208    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-root\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.379284 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.379229    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-node-exporter-tls\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.379284 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.379254    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-metrics-client-ca\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.379284 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.379278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-sys\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.379513 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.379297    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.480520 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.480492    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-root\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.480520 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.480521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-node-exporter-tls\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.480707 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.480546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-metrics-client-ca\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.480707 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.480562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-sys\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.480707 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.480579    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.480707 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.480612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-node-exporter-accelerators-collector-config\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.480707 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.480613    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-root\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.480707 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.480637    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-node-exporter-textfile\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.480707 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.480651    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-sys\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.480707 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:14:35.480659    2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 00:14:35.480707 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.480689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-node-exporter-wtmp\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.481052 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.480716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf6vc\" (UniqueName: \"kubernetes.io/projected/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-kube-api-access-lf6vc\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.481052 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:14:35.480741    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-node-exporter-tls podName:fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4 nodeName:}" failed. No retries permitted until 2026-09-17 00:14:35.980719241 +0000 UTC m=+288.839212360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-node-exporter-tls") pod "node-exporter-xj545" (UID: "fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4") : secret "node-exporter-tls" not found
Sep 17 00:14:35.481052 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.480930    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-node-exporter-wtmp\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.481052 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.480984    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-node-exporter-textfile\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.481236 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.481162    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-metrics-client-ca\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.481272 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.481248    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-node-exporter-accelerators-collector-config\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.482950 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.482933    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.488885 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.488868    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf6vc\" (UniqueName: \"kubernetes.io/projected/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-kube-api-access-lf6vc\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.685432 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.685380    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zhxfj" event={"ID":"57f31c58-e354-4d56-8b7e-c55015c26176","Type":"ContainerStarted","Data":"3d072af3372deeffbbeacffdef40ceae8a36029e535a5b132ed0a1474f8465e3"}
Sep 17 00:14:35.685611 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.685458    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zhxfj" event={"ID":"57f31c58-e354-4d56-8b7e-c55015c26176","Type":"ContainerStarted","Data":"acd359e270374f952245b60b720f233410b9578267e5a6f8cff85f57b4c0ac9a"}
Sep 17 00:14:35.685611 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.685492    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:14:35.687045 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.687012    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x" event={"ID":"b91229a2-6f57-4a69-af2c-8cd92b0459f0","Type":"ContainerStarted","Data":"ecdf732f7da400a944efbfe3f5cc4920a7ecb330acd3a5163f3b31bfd75ba08d"}
Sep 17 00:14:35.702424 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.702377    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zhxfj" podStartSLOduration=251.867166282 podStartE2EDuration="4m13.70236335s" totalImagesPullingTime="1.835197068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:14:35.701595424 +0000 UTC m=+288.560088587" watchObservedRunningTime="2026-09-17 00:14:35.70236335 +0000 UTC m=+288.560856484"
Sep 17 00:14:35.716706 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.716663    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-cch7x" podStartSLOduration=285.88343189 podStartE2EDuration="4m47.716648832s" totalImagesPullingTime="1.833216942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:14:35.715668468 +0000 UTC m=+288.574161606" watchObservedRunningTime="2026-09-17 00:14:35.716648832 +0000 UTC m=+288.575141982"
Sep 17 00:14:35.985026 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.984942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-node-exporter-tls\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:35.987175 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:35.987144    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4-node-exporter-tls\") pod \"node-exporter-xj545\" (UID: \"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4\") " pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:36.214729 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:36.214690    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xj545"
Sep 17 00:14:36.224630 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:14:36.224601    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe3f2779_f3e8_4d7d_9f08_95fa98e5c6a4.slice/crio-1e2d3a19911357af259a56f269b8edb16a51227e64e840e835b36e1aac5b364f WatchSource:0}: Error finding container 1e2d3a19911357af259a56f269b8edb16a51227e64e840e835b36e1aac5b364f: Status 404 returned error can't find the container with id 1e2d3a19911357af259a56f269b8edb16a51227e64e840e835b36e1aac5b364f
Sep 17 00:14:36.692163 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:36.692132    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xj545" event={"ID":"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4","Type":"ContainerStarted","Data":"1e2d3a19911357af259a56f269b8edb16a51227e64e840e835b36e1aac5b364f"}
Sep 17 00:14:37.699822 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:37.699794    2577 generic.go:356] "Generic (PLEG): container finished" podID="33131f95-e9e8-4d86-a9d8-73ff5abb5396" containerID="0c7b359582245fb34f4f272619363cd9e5c1d7b50771b4bca4059f578cc319c7" exitCode=255
Sep 17 00:14:37.700230 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:37.699865    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69897745dd-nhpt7" event={"ID":"33131f95-e9e8-4d86-a9d8-73ff5abb5396","Type":"ContainerDied","Data":"0c7b359582245fb34f4f272619363cd9e5c1d7b50771b4bca4059f578cc319c7"}
Sep 17 00:14:37.700230 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:37.699913    2577 scope.go:117] "RemoveContainer" containerID="265b4933a00f21a14704e18f9efd5e0fa53b5ebd20cf53c2ff60a58da3573fe8"
Sep 17 00:14:37.700353 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:37.700231    2577 scope.go:117] "RemoveContainer" containerID="0c7b359582245fb34f4f272619363cd9e5c1d7b50771b4bca4059f578cc319c7"
Sep 17 00:14:37.700442 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:14:37.700403    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-69897745dd-nhpt7_open-cluster-management-agent-addon(33131f95-e9e8-4d86-a9d8-73ff5abb5396)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69897745dd-nhpt7" podUID="33131f95-e9e8-4d86-a9d8-73ff5abb5396"
Sep 17 00:14:37.701553 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:37.701534    2577 generic.go:356] "Generic (PLEG): container finished" podID="fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4" containerID="3903e39cfbf2d7102ceab11f3ce756f955aae012f694006270ea8688a102464f" exitCode=0
Sep 17 00:14:37.701633 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:37.701598    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xj545" event={"ID":"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4","Type":"ContainerDied","Data":"3903e39cfbf2d7102ceab11f3ce756f955aae012f694006270ea8688a102464f"}
Sep 17 00:14:37.703516 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:37.703499    2577 generic.go:356] "Generic (PLEG): container finished" podID="55925826-a2f5-4c85-9fdd-66e886482069" containerID="49ee82279ea218f412a758a5fe27deb18e3f6a7b3342ee834aa0dc495673c0ef" exitCode=1
Sep 17 00:14:37.703615 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:37.703528    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj" event={"ID":"55925826-a2f5-4c85-9fdd-66e886482069","Type":"ContainerDied","Data":"49ee82279ea218f412a758a5fe27deb18e3f6a7b3342ee834aa0dc495673c0ef"}
Sep 17 00:14:37.703760 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:37.703748    2577 scope.go:117] "RemoveContainer" containerID="49ee82279ea218f412a758a5fe27deb18e3f6a7b3342ee834aa0dc495673c0ef"
Sep 17 00:14:37.703886 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:14:37.703872    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-cc45f99fb-sm5gj_open-cluster-management-agent-addon(55925826-a2f5-4c85-9fdd-66e886482069)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj" podUID="55925826-a2f5-4c85-9fdd-66e886482069"
Sep 17 00:14:37.709380 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:37.709359    2577 scope.go:117] "RemoveContainer" containerID="33e9378bb4983c5de313835b8b691e22ebbd42556a8d9a5f9747d29a6b268ace"
Sep 17 00:14:38.335287 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:38.335254    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj"
Sep 17 00:14:38.710101 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:38.710071    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xj545" event={"ID":"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4","Type":"ContainerStarted","Data":"562d81e03db549f6f739ee9b1a749ea63b668140728744b811972b549c9faa9a"}
Sep 17 00:14:38.710549 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:38.710120    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xj545" event={"ID":"fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4","Type":"ContainerStarted","Data":"5ce727f9b8d69b2ddd7575956e1f69b207dfcdd75b95f6f69e41d12dae75a378"}
Sep 17 00:14:38.711772 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:38.711756    2577 scope.go:117] "RemoveContainer" containerID="49ee82279ea218f412a758a5fe27deb18e3f6a7b3342ee834aa0dc495673c0ef"
Sep 17 00:14:38.711918 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:14:38.711901    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-cc45f99fb-sm5gj_open-cluster-management-agent-addon(55925826-a2f5-4c85-9fdd-66e886482069)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj" podUID="55925826-a2f5-4c85-9fdd-66e886482069"
Sep 17 00:14:38.728651 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:38.728608    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xj545" podStartSLOduration=2.723156228 podStartE2EDuration="3.728594004s" totalImagesPullingTime="944.112982ms" totalInitContainerRuntime="61.324794ms" isStatefulPod=true podCreationTimestamp="2026-09-17 00:14:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:14:38.727036838 +0000 UTC m=+291.585529965" watchObservedRunningTime="2026-09-17 00:14:38.728594004 +0000 UTC m=+291.587087139"
Sep 17 00:14:39.178114 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:39.178018    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69897745dd-nhpt7"
Sep 17 00:14:39.178359 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:39.178345    2577 scope.go:117] "RemoveContainer" containerID="0c7b359582245fb34f4f272619363cd9e5c1d7b50771b4bca4059f578cc319c7"
Sep 17 00:14:39.178541 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:14:39.178522    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-69897745dd-nhpt7_open-cluster-management-agent-addon(33131f95-e9e8-4d86-a9d8-73ff5abb5396)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69897745dd-nhpt7" podUID="33131f95-e9e8-4d86-a9d8-73ff5abb5396"
Sep 17 00:14:39.185350 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:39.185336    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj"
Sep 17 00:14:39.714814 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:39.714788    2577 scope.go:117] "RemoveContainer" containerID="49ee82279ea218f412a758a5fe27deb18e3f6a7b3342ee834aa0dc495673c0ef"
Sep 17 00:14:39.715176 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:14:39.714930    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-cc45f99fb-sm5gj_open-cluster-management-agent-addon(55925826-a2f5-4c85-9fdd-66e886482069)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj" podUID="55925826-a2f5-4c85-9fdd-66e886482069"
Sep 17 00:14:41.602793 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:14:41.602753    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-xbhz5" podUID="b956a62e-c4ac-4e1d-aeb0-3f7907a0a702"
Sep 17 00:14:45.694157 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:45.694130    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zhxfj"
Sep 17 00:14:47.517719 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:47.517691    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/ovn-acl-logging/0.log"
Sep 17 00:14:47.519860 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:47.519839    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/ovn-acl-logging/0.log"
Sep 17 00:14:47.520908 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:47.520888    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 00:14:50.764164 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:50.764128    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5957bbc947-plqh5"]
Sep 17 00:14:50.768445 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:50.768401    2577 patch_prober.go:28] interesting pod/image-registry-5957bbc947-plqh5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 00:14:50.768582 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:50.768477    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5957bbc947-plqh5" podUID="5e823b26-89d1-4a1d-95b0-1d0002258005" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 00:14:51.601744 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:51.601707    2577 scope.go:117] "RemoveContainer" containerID="0c7b359582245fb34f4f272619363cd9e5c1d7b50771b4bca4059f578cc319c7"
Sep 17 00:14:51.755007 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:51.754973    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69897745dd-nhpt7" event={"ID":"33131f95-e9e8-4d86-a9d8-73ff5abb5396","Type":"ContainerStarted","Data":"41980ebe28c3da17a0cc38262ec653badb8c3ce44671c298720f65862f6b0596"}
Sep 17 00:14:54.601101 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:54.601048    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:14:54.601633 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:54.601378    2577 scope.go:117] "RemoveContainer" containerID="49ee82279ea218f412a758a5fe27deb18e3f6a7b3342ee834aa0dc495673c0ef"
Sep 17 00:14:54.766267 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:54.766239    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj" event={"ID":"55925826-a2f5-4c85-9fdd-66e886482069","Type":"ContainerStarted","Data":"60e174390785277cb29055af7020cef860e636b6bf1baefa30d7022eb1d55401"}
Sep 17 00:14:54.766541 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:54.766518    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj"
Sep 17 00:14:54.767926 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:54.767900    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc45f99fb-sm5gj"
Sep 17 00:14:59.221872 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:14:59.221837    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474" podUID="84a8dad6-c057-4734-a357-421d90017acd" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 00:15:00.768925 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:00.768891    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:15:09.222023 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:09.221987    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474" podUID="84a8dad6-c057-4734-a357-421d90017acd" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 00:15:15.782492 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:15.782424    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5957bbc947-plqh5" podUID="5e823b26-89d1-4a1d-95b0-1d0002258005" containerName="registry" containerID="cri-o://a0bff7f87e60e1e5d5a4d28a6d8b43bab797b30aa3eef2cf053870a53ab1965b" gracePeriod=30
Sep 17 00:15:16.020957 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.020934    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:15:16.196600 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.196567    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-858l4\" (UniqueName: \"kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-kube-api-access-858l4\") pod \"5e823b26-89d1-4a1d-95b0-1d0002258005\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") "
Sep 17 00:15:16.196782 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.196627    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-bound-sa-token\") pod \"5e823b26-89d1-4a1d-95b0-1d0002258005\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") "
Sep 17 00:15:16.196782 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.196653    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e823b26-89d1-4a1d-95b0-1d0002258005-ca-trust-extracted\") pod \"5e823b26-89d1-4a1d-95b0-1d0002258005\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") "
Sep 17 00:15:16.196782 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.196682    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e823b26-89d1-4a1d-95b0-1d0002258005-trusted-ca\") pod \"5e823b26-89d1-4a1d-95b0-1d0002258005\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") "
Sep 17 00:15:16.196782 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.196703    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-certificates\") pod \"5e823b26-89d1-4a1d-95b0-1d0002258005\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") "
Sep 17 00:15:16.196973 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.196802    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e823b26-89d1-4a1d-95b0-1d0002258005-installation-pull-secrets\") pod \"5e823b26-89d1-4a1d-95b0-1d0002258005\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") "
Sep 17 00:15:16.196973 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.196877    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls\") pod \"5e823b26-89d1-4a1d-95b0-1d0002258005\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") "
Sep 17 00:15:16.196973 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.196911    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e823b26-89d1-4a1d-95b0-1d0002258005-image-registry-private-configuration\") pod \"5e823b26-89d1-4a1d-95b0-1d0002258005\" (UID: \"5e823b26-89d1-4a1d-95b0-1d0002258005\") "
Sep 17 00:15:16.197255 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.197221    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e823b26-89d1-4a1d-95b0-1d0002258005-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5e823b26-89d1-4a1d-95b0-1d0002258005" (UID: "5e823b26-89d1-4a1d-95b0-1d0002258005"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:15:16.197620 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.197579    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5e823b26-89d1-4a1d-95b0-1d0002258005" (UID: "5e823b26-89d1-4a1d-95b0-1d0002258005"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:15:16.199206 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.199142    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5e823b26-89d1-4a1d-95b0-1d0002258005" (UID: "5e823b26-89d1-4a1d-95b0-1d0002258005"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:15:16.199401 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.199378    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e823b26-89d1-4a1d-95b0-1d0002258005-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5e823b26-89d1-4a1d-95b0-1d0002258005" (UID: "5e823b26-89d1-4a1d-95b0-1d0002258005"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:15:16.199492 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.199391    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-kube-api-access-858l4" (OuterVolumeSpecName: "kube-api-access-858l4") pod "5e823b26-89d1-4a1d-95b0-1d0002258005" (UID: "5e823b26-89d1-4a1d-95b0-1d0002258005"). InnerVolumeSpecName "kube-api-access-858l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:15:16.199594 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.199573    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e823b26-89d1-4a1d-95b0-1d0002258005-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5e823b26-89d1-4a1d-95b0-1d0002258005" (UID: "5e823b26-89d1-4a1d-95b0-1d0002258005"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:15:16.199670 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.199651    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5e823b26-89d1-4a1d-95b0-1d0002258005" (UID: "5e823b26-89d1-4a1d-95b0-1d0002258005"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:15:16.208601 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.208573    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e823b26-89d1-4a1d-95b0-1d0002258005-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5e823b26-89d1-4a1d-95b0-1d0002258005" (UID: "5e823b26-89d1-4a1d-95b0-1d0002258005"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:15:16.298045 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.298012    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-858l4\" (UniqueName: \"kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-kube-api-access-858l4\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:15:16.298045 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.298038    2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-bound-sa-token\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:15:16.298045 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.298049    2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e823b26-89d1-4a1d-95b0-1d0002258005-ca-trust-extracted\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:15:16.298244 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.298057    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e823b26-89d1-4a1d-95b0-1d0002258005-trusted-ca\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:15:16.298244 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.298065    2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-certificates\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:15:16.298244 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.298072    2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e823b26-89d1-4a1d-95b0-1d0002258005-installation-pull-secrets\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:15:16.298244 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.298080    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e823b26-89d1-4a1d-95b0-1d0002258005-registry-tls\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:15:16.298244 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.298089    2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e823b26-89d1-4a1d-95b0-1d0002258005-image-registry-private-configuration\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:15:16.840781 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.840746    2577 generic.go:356] "Generic (PLEG): container finished" podID="5e823b26-89d1-4a1d-95b0-1d0002258005" containerID="a0bff7f87e60e1e5d5a4d28a6d8b43bab797b30aa3eef2cf053870a53ab1965b" exitCode=0
Sep 17 00:15:16.841185 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.840790    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5957bbc947-plqh5"
Sep 17 00:15:16.841185 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.840808    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5957bbc947-plqh5" event={"ID":"5e823b26-89d1-4a1d-95b0-1d0002258005","Type":"ContainerDied","Data":"a0bff7f87e60e1e5d5a4d28a6d8b43bab797b30aa3eef2cf053870a53ab1965b"}
Sep 17 00:15:16.841185 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.840856    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5957bbc947-plqh5" event={"ID":"5e823b26-89d1-4a1d-95b0-1d0002258005","Type":"ContainerDied","Data":"7bbec06d0bc7b189160710d3378bb054012d931c8d167c3637753781e9cbb040"}
Sep 17 00:15:16.841185 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.840875    2577 scope.go:117] "RemoveContainer" containerID="a0bff7f87e60e1e5d5a4d28a6d8b43bab797b30aa3eef2cf053870a53ab1965b"
Sep 17 00:15:16.853082 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.853062    2577 scope.go:117] "RemoveContainer" containerID="a0bff7f87e60e1e5d5a4d28a6d8b43bab797b30aa3eef2cf053870a53ab1965b"
Sep 17 00:15:16.853365 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:15:16.853345    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0bff7f87e60e1e5d5a4d28a6d8b43bab797b30aa3eef2cf053870a53ab1965b\": container with ID starting with a0bff7f87e60e1e5d5a4d28a6d8b43bab797b30aa3eef2cf053870a53ab1965b not found: ID does not exist" containerID="a0bff7f87e60e1e5d5a4d28a6d8b43bab797b30aa3eef2cf053870a53ab1965b"
Sep 17 00:15:16.853449 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.853377    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0bff7f87e60e1e5d5a4d28a6d8b43bab797b30aa3eef2cf053870a53ab1965b"} err="failed to get container status \"a0bff7f87e60e1e5d5a4d28a6d8b43bab797b30aa3eef2cf053870a53ab1965b\": rpc error: code = NotFound desc = could not find container \"a0bff7f87e60e1e5d5a4d28a6d8b43bab797b30aa3eef2cf053870a53ab1965b\": container with ID starting with a0bff7f87e60e1e5d5a4d28a6d8b43bab797b30aa3eef2cf053870a53ab1965b not found: ID does not exist"
Sep 17 00:15:16.864242 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.864220    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5957bbc947-plqh5"]
Sep 17 00:15:16.869850 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:16.869830    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5957bbc947-plqh5"]
Sep 17 00:15:17.606423 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:17.606393    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e823b26-89d1-4a1d-95b0-1d0002258005" path="/var/lib/kubelet/pods/5e823b26-89d1-4a1d-95b0-1d0002258005/volumes"
Sep 17 00:15:19.222171 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:19.222138    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474" podUID="84a8dad6-c057-4734-a357-421d90017acd" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 00:15:19.222575 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:19.222201    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474"
Sep 17 00:15:19.222694 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:19.222677    2577 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"f34b393fd0aff90f1dc4f278a6c12d27a736cd2e24726c7e0fd43dfffdb292de"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 17 00:15:19.222736 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:19.222717    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474" podUID="84a8dad6-c057-4734-a357-421d90017acd" containerName="service-proxy" containerID="cri-o://f34b393fd0aff90f1dc4f278a6c12d27a736cd2e24726c7e0fd43dfffdb292de" gracePeriod=30
Sep 17 00:15:19.852497 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:19.852466    2577 generic.go:356] "Generic (PLEG): container finished" podID="84a8dad6-c057-4734-a357-421d90017acd" containerID="f34b393fd0aff90f1dc4f278a6c12d27a736cd2e24726c7e0fd43dfffdb292de" exitCode=2
Sep 17 00:15:19.852671 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:19.852524    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474" event={"ID":"84a8dad6-c057-4734-a357-421d90017acd","Type":"ContainerDied","Data":"f34b393fd0aff90f1dc4f278a6c12d27a736cd2e24726c7e0fd43dfffdb292de"}
Sep 17 00:15:19.852671 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:15:19.852559    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d65658c7-qq474" event={"ID":"84a8dad6-c057-4734-a357-421d90017acd","Type":"ContainerStarted","Data":"24edc76f5915f8df311e669ff840b75da1207e46bbef1bf2082002f30b36d96e"}
Sep 17 00:16:00.621795 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:00.621748    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs\") pod \"network-metrics-daemon-xbhz5\" (UID: \"b956a62e-c4ac-4e1d-aeb0-3f7907a0a702\") " pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:16:00.624039 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:00.624011    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b956a62e-c4ac-4e1d-aeb0-3f7907a0a702-metrics-certs\") pod \"network-metrics-daemon-xbhz5\" (UID: \"b956a62e-c4ac-4e1d-aeb0-3f7907a0a702\") " pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:16:00.905104 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:00.905020    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zxlvp\""
Sep 17 00:16:00.912993 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:00.912965    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbhz5"
Sep 17 00:16:01.183264 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:01.183231    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xbhz5"]
Sep 17 00:16:01.186299 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:16:01.186273    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb956a62e_c4ac_4e1d_aeb0_3f7907a0a702.slice/crio-622629e6b05b47a52874d67b86705121226f6caeb5095ab6a49592b97d170a18 WatchSource:0}: Error finding container 622629e6b05b47a52874d67b86705121226f6caeb5095ab6a49592b97d170a18: Status 404 returned error can't find the container with id 622629e6b05b47a52874d67b86705121226f6caeb5095ab6a49592b97d170a18
Sep 17 00:16:01.190628 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:01.190602    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 00:16:01.986522 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:01.986473    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xbhz5" event={"ID":"b956a62e-c4ac-4e1d-aeb0-3f7907a0a702","Type":"ContainerStarted","Data":"622629e6b05b47a52874d67b86705121226f6caeb5095ab6a49592b97d170a18"}
Sep 17 00:16:02.991292 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:02.991249    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xbhz5" event={"ID":"b956a62e-c4ac-4e1d-aeb0-3f7907a0a702","Type":"ContainerStarted","Data":"4a7b425175ead553e3b25702c6c35e6a3252adf67e58acca396033f8a1a149f7"}
Sep 17 00:16:02.991729 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:02.991316    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xbhz5" event={"ID":"b956a62e-c4ac-4e1d-aeb0-3f7907a0a702","Type":"ContainerStarted","Data":"76f7aeabaf92c754abd487e396d0388bd3878b79d317facb756ed9b1dd0dc20f"}
Sep 17 00:16:03.007373 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:03.007319    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xbhz5" podStartSLOduration=373.789252263 podStartE2EDuration="6m15.007298759s" totalImagesPullingTime="1.218046496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:16:03.006693443 +0000 UTC m=+375.865186579" watchObservedRunningTime="2026-09-17 00:16:03.007298759 +0000 UTC m=+375.865791895"
Sep 17 00:16:41.184518 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:41.184483    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz"]
Sep 17 00:16:41.185093 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:41.184818    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e823b26-89d1-4a1d-95b0-1d0002258005" containerName="registry"
Sep 17 00:16:41.185093 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:41.184833    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e823b26-89d1-4a1d-95b0-1d0002258005" containerName="registry"
Sep 17 00:16:41.185093 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:41.184912    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e823b26-89d1-4a1d-95b0-1d0002258005" containerName="registry"
Sep 17 00:16:41.187603 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:41.187578    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz"
Sep 17 00:16:41.190199 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:41.190168    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 00:16:41.190199 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:41.190168    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-q8lnv\""
Sep 17 00:16:41.190387 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:41.190227    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 00:16:41.195658 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:41.195634    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz"]
Sep 17 00:16:41.306111 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:41.306067    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d2b4550-a2fb-4204-a7b9-22b2b61e8220-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz\" (UID: \"2d2b4550-a2fb-4204-a7b9-22b2b61e8220\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz"
Sep 17 00:16:41.306111 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:41.306108    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xgtm\" (UniqueName: \"kubernetes.io/projected/2d2b4550-a2fb-4204-a7b9-22b2b61e8220-kube-api-access-8xgtm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz\" (UID: \"2d2b4550-a2fb-4204-a7b9-22b2b61e8220\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz"
Sep 17 00:16:41.306345 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:41.306145    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d2b4550-a2fb-4204-a7b9-22b2b61e8220-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz\" (UID: \"2d2b4550-a2fb-4204-a7b9-22b2b61e8220\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz"
Sep 17 00:16:41.406917 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:41.406886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d2b4550-a2fb-4204-a7b9-22b2b61e8220-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz\" (UID: \"2d2b4550-a2fb-4204-a7b9-22b2b61e8220\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz"
Sep 17 00:16:41.407084 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:41.406940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d2b4550-a2fb-4204-a7b9-22b2b61e8220-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz\" (UID: \"2d2b4550-a2fb-4204-a7b9-22b2b61e8220\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz"
Sep 17 00:16:41.407084 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:41.407039    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xgtm\" (UniqueName: \"kubernetes.io/projected/2d2b4550-a2fb-4204-a7b9-22b2b61e8220-kube-api-access-8xgtm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz\" (UID: \"2d2b4550-a2fb-4204-a7b9-22b2b61e8220\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz"
Sep 17 00:16:41.407250 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:41.407235    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d2b4550-a2fb-4204-a7b9-22b2b61e8220-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz\" (UID: \"2d2b4550-a2fb-4204-a7b9-22b2b61e8220\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz"
Sep 17 00:16:41.407300 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:41.407283    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d2b4550-a2fb-4204-a7b9-22b2b61e8220-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz\" (UID: \"2d2b4550-a2fb-4204-a7b9-22b2b61e8220\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz"
Sep 17 00:16:41.415430 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:41.415393    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xgtm\" (UniqueName: \"kubernetes.io/projected/2d2b4550-a2fb-4204-a7b9-22b2b61e8220-kube-api-access-8xgtm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz\" (UID: \"2d2b4550-a2fb-4204-a7b9-22b2b61e8220\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz"
Sep 17 00:16:41.496583 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:41.496506    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz"
Sep 17 00:16:41.764776 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:41.764709    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz"]
Sep 17 00:16:41.767728 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:16:41.767696    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2b4550_a2fb_4204_a7b9_22b2b61e8220.slice/crio-af43bebd5f21427c437b42e8553af27e7ef2ac99005af274b9ee4023e35ea4aa WatchSource:0}: Error finding container af43bebd5f21427c437b42e8553af27e7ef2ac99005af274b9ee4023e35ea4aa: Status 404 returned error can't find the container with id af43bebd5f21427c437b42e8553af27e7ef2ac99005af274b9ee4023e35ea4aa
Sep 17 00:16:42.118114 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:42.118031    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz" event={"ID":"2d2b4550-a2fb-4204-a7b9-22b2b61e8220","Type":"ContainerStarted","Data":"af43bebd5f21427c437b42e8553af27e7ef2ac99005af274b9ee4023e35ea4aa"}
Sep 17 00:16:48.144824 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:48.144787    2577 generic.go:356] "Generic (PLEG): container finished" podID="2d2b4550-a2fb-4204-a7b9-22b2b61e8220" containerID="e97ce8b977374a9dee99c5ce0cf2e75e9232ad8d60b98262e367c22007f7f95e" exitCode=0
Sep 17 00:16:48.145206 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:48.144879    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz" event={"ID":"2d2b4550-a2fb-4204-a7b9-22b2b61e8220","Type":"ContainerDied","Data":"e97ce8b977374a9dee99c5ce0cf2e75e9232ad8d60b98262e367c22007f7f95e"}
Sep 17 00:16:52.160023 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:52.159990    2577 generic.go:356] "Generic (PLEG): container finished" podID="2d2b4550-a2fb-4204-a7b9-22b2b61e8220" containerID="16c842f981c1c9cec9ab4304ea13c6198edb242dabda60cd45afa3a2b1f25e45" exitCode=0
Sep 17 00:16:52.160410 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:52.160048    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz" event={"ID":"2d2b4550-a2fb-4204-a7b9-22b2b61e8220","Type":"ContainerDied","Data":"16c842f981c1c9cec9ab4304ea13c6198edb242dabda60cd45afa3a2b1f25e45"}
Sep 17 00:16:59.189862 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:59.189827    2577 generic.go:356] "Generic (PLEG): container finished" podID="2d2b4550-a2fb-4204-a7b9-22b2b61e8220" containerID="100ffba58200230e1b4fda42afd6b8198f4e9b09060efdb559f1bc14848e158e" exitCode=0
Sep 17 00:16:59.190293 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:16:59.189869    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz" event={"ID":"2d2b4550-a2fb-4204-a7b9-22b2b61e8220","Type":"ContainerDied","Data":"100ffba58200230e1b4fda42afd6b8198f4e9b09060efdb559f1bc14848e158e"}
Sep 17 00:17:00.320237 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:00.320214    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz"
Sep 17 00:17:00.456738 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:00.456637    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xgtm\" (UniqueName: \"kubernetes.io/projected/2d2b4550-a2fb-4204-a7b9-22b2b61e8220-kube-api-access-8xgtm\") pod \"2d2b4550-a2fb-4204-a7b9-22b2b61e8220\" (UID: \"2d2b4550-a2fb-4204-a7b9-22b2b61e8220\") "
Sep 17 00:17:00.456738 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:00.456698    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d2b4550-a2fb-4204-a7b9-22b2b61e8220-util\") pod \"2d2b4550-a2fb-4204-a7b9-22b2b61e8220\" (UID: \"2d2b4550-a2fb-4204-a7b9-22b2b61e8220\") "
Sep 17 00:17:00.456738 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:00.456721    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d2b4550-a2fb-4204-a7b9-22b2b61e8220-bundle\") pod \"2d2b4550-a2fb-4204-a7b9-22b2b61e8220\" (UID: \"2d2b4550-a2fb-4204-a7b9-22b2b61e8220\") "
Sep 17 00:17:00.457330 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:00.457302    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2b4550-a2fb-4204-a7b9-22b2b61e8220-bundle" (OuterVolumeSpecName: "bundle") pod "2d2b4550-a2fb-4204-a7b9-22b2b61e8220" (UID: "2d2b4550-a2fb-4204-a7b9-22b2b61e8220"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:17:00.458844 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:00.458819    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2b4550-a2fb-4204-a7b9-22b2b61e8220-kube-api-access-8xgtm" (OuterVolumeSpecName: "kube-api-access-8xgtm") pod "2d2b4550-a2fb-4204-a7b9-22b2b61e8220" (UID: "2d2b4550-a2fb-4204-a7b9-22b2b61e8220"). InnerVolumeSpecName "kube-api-access-8xgtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:17:00.460854 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:00.460830    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2b4550-a2fb-4204-a7b9-22b2b61e8220-util" (OuterVolumeSpecName: "util") pod "2d2b4550-a2fb-4204-a7b9-22b2b61e8220" (UID: "2d2b4550-a2fb-4204-a7b9-22b2b61e8220"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:17:00.557943 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:00.557907    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xgtm\" (UniqueName: \"kubernetes.io/projected/2d2b4550-a2fb-4204-a7b9-22b2b61e8220-kube-api-access-8xgtm\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:17:00.557943 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:00.557936    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d2b4550-a2fb-4204-a7b9-22b2b61e8220-util\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:17:00.557943 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:00.557946    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d2b4550-a2fb-4204-a7b9-22b2b61e8220-bundle\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:17:01.197917 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:01.197883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz" event={"ID":"2d2b4550-a2fb-4204-a7b9-22b2b61e8220","Type":"ContainerDied","Data":"af43bebd5f21427c437b42e8553af27e7ef2ac99005af274b9ee4023e35ea4aa"}
Sep 17 00:17:01.198090 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:01.197924    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qldpz"
Sep 17 00:17:01.198090 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:01.197932    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af43bebd5f21427c437b42e8553af27e7ef2ac99005af274b9ee4023e35ea4aa"
Sep 17 00:17:14.096206 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:14.096117    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r"]
Sep 17 00:17:14.096779 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:14.096473    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d2b4550-a2fb-4204-a7b9-22b2b61e8220" containerName="util"
Sep 17 00:17:14.096779 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:14.096493    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d2b4550-a2fb-4204-a7b9-22b2b61e8220" containerName="util"
Sep 17 00:17:14.096779 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:14.096505    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d2b4550-a2fb-4204-a7b9-22b2b61e8220" containerName="pull"
Sep 17 00:17:14.096779 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:14.096513    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d2b4550-a2fb-4204-a7b9-22b2b61e8220" containerName="pull"
Sep 17 00:17:14.096779 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:14.096522    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d2b4550-a2fb-4204-a7b9-22b2b61e8220" containerName="extract"
Sep 17 00:17:14.096779 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:14.096531    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d2b4550-a2fb-4204-a7b9-22b2b61e8220" containerName="extract"
Sep 17 00:17:14.096779 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:14.096596    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d2b4550-a2fb-4204-a7b9-22b2b61e8220" containerName="extract"
Sep 17 00:17:14.098740 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:14.098720    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r"
Sep 17 00:17:14.101255 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:14.101226    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 00:17:14.101360 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:14.101295    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 00:17:14.102278 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:14.102261    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-q8lnv\""
Sep 17 00:17:14.108220 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:14.108194    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r"]
Sep 17 00:17:14.149029 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:14.148984    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4356ec8-6eca-4885-9044-86cddf383484-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r\" (UID: \"c4356ec8-6eca-4885-9044-86cddf383484\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r"
Sep 17 00:17:14.149029 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:14.149027    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh46m\" (UniqueName: \"kubernetes.io/projected/c4356ec8-6eca-4885-9044-86cddf383484-kube-api-access-kh46m\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r\" (UID: \"c4356ec8-6eca-4885-9044-86cddf383484\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r"
Sep 17 00:17:14.149239 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:14.149124    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4356ec8-6eca-4885-9044-86cddf383484-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r\" (UID: \"c4356ec8-6eca-4885-9044-86cddf383484\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r"
Sep 17 00:17:14.249609 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:14.249576    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4356ec8-6eca-4885-9044-86cddf383484-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r\" (UID: \"c4356ec8-6eca-4885-9044-86cddf383484\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r"
Sep 17 00:17:14.249761 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:14.249619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4356ec8-6eca-4885-9044-86cddf383484-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r\" (UID: \"c4356ec8-6eca-4885-9044-86cddf383484\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r"
Sep 17 00:17:14.249761 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:14.249643    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh46m\" (UniqueName: \"kubernetes.io/projected/c4356ec8-6eca-4885-9044-86cddf383484-kube-api-access-kh46m\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r\" (UID: \"c4356ec8-6eca-4885-9044-86cddf383484\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r"
Sep 17 00:17:14.249959 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:14.249938    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4356ec8-6eca-4885-9044-86cddf383484-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r\" (UID: \"c4356ec8-6eca-4885-9044-86cddf383484\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r"
Sep 17 00:17:14.249999 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:14.249977    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4356ec8-6eca-4885-9044-86cddf383484-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r\" (UID: \"c4356ec8-6eca-4885-9044-86cddf383484\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r"
Sep 17 00:17:14.261142 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:14.261122    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh46m\" (UniqueName: \"kubernetes.io/projected/c4356ec8-6eca-4885-9044-86cddf383484-kube-api-access-kh46m\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r\" (UID: \"c4356ec8-6eca-4885-9044-86cddf383484\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r"
Sep 17 00:17:14.408336 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:14.408247    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r"
Sep 17 00:17:14.676821 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:14.676732    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r"]
Sep 17 00:17:14.679601 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:17:14.679576    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4356ec8_6eca_4885_9044_86cddf383484.slice/crio-4e85df69abc9af0f13cc49920ee2a4df83ca398514c9422971aadcc70f6920f2 WatchSource:0}: Error finding container 4e85df69abc9af0f13cc49920ee2a4df83ca398514c9422971aadcc70f6920f2: Status 404 returned error can't find the container with id 4e85df69abc9af0f13cc49920ee2a4df83ca398514c9422971aadcc70f6920f2
Sep 17 00:17:15.244102 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:15.244067    2577 generic.go:356] "Generic (PLEG): container finished" podID="c4356ec8-6eca-4885-9044-86cddf383484" containerID="63d4f3b2c459b4747c78beaab3d7005dfe80b89f120f3a5cdcbf0a40aa332119" exitCode=0
Sep 17 00:17:15.244550 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:15.244143    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r" event={"ID":"c4356ec8-6eca-4885-9044-86cddf383484","Type":"ContainerDied","Data":"63d4f3b2c459b4747c78beaab3d7005dfe80b89f120f3a5cdcbf0a40aa332119"}
Sep 17 00:17:15.244550 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:15.244202    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r" event={"ID":"c4356ec8-6eca-4885-9044-86cddf383484","Type":"ContainerStarted","Data":"4e85df69abc9af0f13cc49920ee2a4df83ca398514c9422971aadcc70f6920f2"}
Sep 17 00:17:18.256920 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:18.256884    2577 generic.go:356] "Generic (PLEG): container finished" podID="c4356ec8-6eca-4885-9044-86cddf383484" containerID="3d3aeaf882efe16a0b458986e3be6f2ef3ae2ea4031d56c48f930ca1f3b4bbea" exitCode=0
Sep 17 00:17:18.257294 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:18.256934    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r" event={"ID":"c4356ec8-6eca-4885-9044-86cddf383484","Type":"ContainerDied","Data":"3d3aeaf882efe16a0b458986e3be6f2ef3ae2ea4031d56c48f930ca1f3b4bbea"}
Sep 17 00:17:19.262402 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:19.262373    2577 generic.go:356] "Generic (PLEG): container finished" podID="c4356ec8-6eca-4885-9044-86cddf383484" containerID="fac3620f86b2f5c7cdd5273f8848bcf567b8e56f683ce5eb719f54fc344ecc68" exitCode=0
Sep 17 00:17:19.262811 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:19.262465    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r" event={"ID":"c4356ec8-6eca-4885-9044-86cddf383484","Type":"ContainerDied","Data":"fac3620f86b2f5c7cdd5273f8848bcf567b8e56f683ce5eb719f54fc344ecc68"}
Sep 17 00:17:20.382854 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:20.382831    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r"
Sep 17 00:17:20.394946 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:20.394929    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4356ec8-6eca-4885-9044-86cddf383484-util\") pod \"c4356ec8-6eca-4885-9044-86cddf383484\" (UID: \"c4356ec8-6eca-4885-9044-86cddf383484\") "
Sep 17 00:17:20.395028 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:20.394989    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4356ec8-6eca-4885-9044-86cddf383484-bundle\") pod \"c4356ec8-6eca-4885-9044-86cddf383484\" (UID: \"c4356ec8-6eca-4885-9044-86cddf383484\") "
Sep 17 00:17:20.395028 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:20.395015    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kh46m\" (UniqueName: \"kubernetes.io/projected/c4356ec8-6eca-4885-9044-86cddf383484-kube-api-access-kh46m\") pod \"c4356ec8-6eca-4885-9044-86cddf383484\" (UID: \"c4356ec8-6eca-4885-9044-86cddf383484\") "
Sep 17 00:17:20.395377 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:20.395353    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4356ec8-6eca-4885-9044-86cddf383484-bundle" (OuterVolumeSpecName: "bundle") pod "c4356ec8-6eca-4885-9044-86cddf383484" (UID: "c4356ec8-6eca-4885-9044-86cddf383484"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:17:20.396951 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:20.396926    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4356ec8-6eca-4885-9044-86cddf383484-kube-api-access-kh46m" (OuterVolumeSpecName: "kube-api-access-kh46m") pod "c4356ec8-6eca-4885-9044-86cddf383484" (UID: "c4356ec8-6eca-4885-9044-86cddf383484"). InnerVolumeSpecName "kube-api-access-kh46m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:17:20.399685 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:20.399654    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4356ec8-6eca-4885-9044-86cddf383484-util" (OuterVolumeSpecName: "util") pod "c4356ec8-6eca-4885-9044-86cddf383484" (UID: "c4356ec8-6eca-4885-9044-86cddf383484"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:17:20.495812 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:20.495762    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4356ec8-6eca-4885-9044-86cddf383484-bundle\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:17:20.495812 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:20.495805    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh46m\" (UniqueName: \"kubernetes.io/projected/c4356ec8-6eca-4885-9044-86cddf383484-kube-api-access-kh46m\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:17:20.495812 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:20.495817    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4356ec8-6eca-4885-9044-86cddf383484-util\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:17:21.271333 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:21.271297    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r" event={"ID":"c4356ec8-6eca-4885-9044-86cddf383484","Type":"ContainerDied","Data":"4e85df69abc9af0f13cc49920ee2a4df83ca398514c9422971aadcc70f6920f2"}
Sep 17 00:17:21.271501 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:21.271344    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e85df69abc9af0f13cc49920ee2a4df83ca398514c9422971aadcc70f6920f2"
Sep 17 00:17:21.271501 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:21.271311    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgd7r"
Sep 17 00:17:26.367891 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.367859    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kdnhc"]
Sep 17 00:17:26.368241 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.368074    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4356ec8-6eca-4885-9044-86cddf383484" containerName="util"
Sep 17 00:17:26.368241 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.368085    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4356ec8-6eca-4885-9044-86cddf383484" containerName="util"
Sep 17 00:17:26.368241 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.368146    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4356ec8-6eca-4885-9044-86cddf383484" containerName="pull"
Sep 17 00:17:26.368241 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.368151    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4356ec8-6eca-4885-9044-86cddf383484" containerName="pull"
Sep 17 00:17:26.368241 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.368159    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4356ec8-6eca-4885-9044-86cddf383484" containerName="extract"
Sep 17 00:17:26.368241 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.368165    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4356ec8-6eca-4885-9044-86cddf383484" containerName="extract"
Sep 17 00:17:26.368241 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.368203    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4356ec8-6eca-4885-9044-86cddf383484" containerName="extract"
Sep 17 00:17:26.371723 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.371706    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-kdnhc"
Sep 17 00:17:26.374011 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.373988    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 00:17:26.374127 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.374030    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-wzgc2\""
Sep 17 00:17:26.374127 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.374030    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 00:17:26.379152 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.379130    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kdnhc"]
Sep 17 00:17:26.437217 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.437177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b40d7af-5ab8-4d80-9cf3-394b49de9e65-bound-sa-token\") pod \"cert-manager-55b4d954d5-kdnhc\" (UID: \"0b40d7af-5ab8-4d80-9cf3-394b49de9e65\") " pod="cert-manager/cert-manager-55b4d954d5-kdnhc"
Sep 17 00:17:26.437217 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.437220    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jv2k\" (UniqueName: \"kubernetes.io/projected/0b40d7af-5ab8-4d80-9cf3-394b49de9e65-kube-api-access-7jv2k\") pod \"cert-manager-55b4d954d5-kdnhc\" (UID: \"0b40d7af-5ab8-4d80-9cf3-394b49de9e65\") " pod="cert-manager/cert-manager-55b4d954d5-kdnhc"
Sep 17 00:17:26.538388 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.538354    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b40d7af-5ab8-4d80-9cf3-394b49de9e65-bound-sa-token\") pod \"cert-manager-55b4d954d5-kdnhc\" (UID: \"0b40d7af-5ab8-4d80-9cf3-394b49de9e65\") " pod="cert-manager/cert-manager-55b4d954d5-kdnhc"
Sep 17 00:17:26.538388 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.538392    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jv2k\" (UniqueName: \"kubernetes.io/projected/0b40d7af-5ab8-4d80-9cf3-394b49de9e65-kube-api-access-7jv2k\") pod \"cert-manager-55b4d954d5-kdnhc\" (UID: \"0b40d7af-5ab8-4d80-9cf3-394b49de9e65\") " pod="cert-manager/cert-manager-55b4d954d5-kdnhc"
Sep 17 00:17:26.548504 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.548480    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b40d7af-5ab8-4d80-9cf3-394b49de9e65-bound-sa-token\") pod \"cert-manager-55b4d954d5-kdnhc\" (UID: \"0b40d7af-5ab8-4d80-9cf3-394b49de9e65\") " pod="cert-manager/cert-manager-55b4d954d5-kdnhc"
Sep 17 00:17:26.548767 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.548747    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jv2k\" (UniqueName: \"kubernetes.io/projected/0b40d7af-5ab8-4d80-9cf3-394b49de9e65-kube-api-access-7jv2k\") pod \"cert-manager-55b4d954d5-kdnhc\" (UID: \"0b40d7af-5ab8-4d80-9cf3-394b49de9e65\") " pod="cert-manager/cert-manager-55b4d954d5-kdnhc"
Sep 17 00:17:26.658731 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.658653    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vdppl"]
Sep 17 00:17:26.662393 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.662377    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vdppl"
Sep 17 00:17:26.664911 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.664892    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-d9k9z\""
Sep 17 00:17:26.665007 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.664942    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 00:17:26.665233 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.665208    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 00:17:26.673083 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.673057    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vdppl"]
Sep 17 00:17:26.680627 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.680607    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-kdnhc"
Sep 17 00:17:26.740003 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.739975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqvt5\" (UniqueName: \"kubernetes.io/projected/a6ea1ee9-37a6-40e6-bb74-a58075f729d5-kube-api-access-nqvt5\") pod \"openshift-lws-operator-fd8ccff4c-vdppl\" (UID: \"a6ea1ee9-37a6-40e6-bb74-a58075f729d5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vdppl"
Sep 17 00:17:26.740148 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.740134    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6ea1ee9-37a6-40e6-bb74-a58075f729d5-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vdppl\" (UID: \"a6ea1ee9-37a6-40e6-bb74-a58075f729d5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vdppl"
Sep 17 00:17:26.840782 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.840751    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqvt5\" (UniqueName: \"kubernetes.io/projected/a6ea1ee9-37a6-40e6-bb74-a58075f729d5-kube-api-access-nqvt5\") pod \"openshift-lws-operator-fd8ccff4c-vdppl\" (UID: \"a6ea1ee9-37a6-40e6-bb74-a58075f729d5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vdppl"
Sep 17 00:17:26.840782 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.840786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6ea1ee9-37a6-40e6-bb74-a58075f729d5-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vdppl\" (UID: \"a6ea1ee9-37a6-40e6-bb74-a58075f729d5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vdppl"
Sep 17 00:17:26.841114 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.841098    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6ea1ee9-37a6-40e6-bb74-a58075f729d5-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vdppl\" (UID: \"a6ea1ee9-37a6-40e6-bb74-a58075f729d5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vdppl"
Sep 17 00:17:26.848237 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.848206    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqvt5\" (UniqueName: \"kubernetes.io/projected/a6ea1ee9-37a6-40e6-bb74-a58075f729d5-kube-api-access-nqvt5\") pod \"openshift-lws-operator-fd8ccff4c-vdppl\" (UID: \"a6ea1ee9-37a6-40e6-bb74-a58075f729d5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vdppl"
Sep 17 00:17:26.949435 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.949396    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kdnhc"]
Sep 17 00:17:26.951926 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:17:26.951900    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b40d7af_5ab8_4d80_9cf3_394b49de9e65.slice/crio-aba4565a2469600a7e146e1557d6d39de7e29b3b3ac15f66b1820639c920a3b0 WatchSource:0}: Error finding container aba4565a2469600a7e146e1557d6d39de7e29b3b3ac15f66b1820639c920a3b0: Status 404 returned error can't find the container with id aba4565a2469600a7e146e1557d6d39de7e29b3b3ac15f66b1820639c920a3b0
Sep 17 00:17:26.977285 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:26.977255    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vdppl"
Sep 17 00:17:27.248559 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:27.248534    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vdppl"]
Sep 17 00:17:27.250713 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:17:27.250680    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6ea1ee9_37a6_40e6_bb74_a58075f729d5.slice/crio-2ecd6231a488497ddeaf3bbc197f0ed28a48ac19c75c9420636379d679714e50 WatchSource:0}: Error finding container 2ecd6231a488497ddeaf3bbc197f0ed28a48ac19c75c9420636379d679714e50: Status 404 returned error can't find the container with id 2ecd6231a488497ddeaf3bbc197f0ed28a48ac19c75c9420636379d679714e50
Sep 17 00:17:27.293099 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:27.293070    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vdppl" event={"ID":"a6ea1ee9-37a6-40e6-bb74-a58075f729d5","Type":"ContainerStarted","Data":"2ecd6231a488497ddeaf3bbc197f0ed28a48ac19c75c9420636379d679714e50"}
Sep 17 00:17:27.294215 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:27.294186    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-kdnhc" event={"ID":"0b40d7af-5ab8-4d80-9cf3-394b49de9e65","Type":"ContainerStarted","Data":"aba4565a2469600a7e146e1557d6d39de7e29b3b3ac15f66b1820639c920a3b0"}
Sep 17 00:17:31.311380 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:31.311337    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vdppl" event={"ID":"a6ea1ee9-37a6-40e6-bb74-a58075f729d5","Type":"ContainerStarted","Data":"939919b31f1c9510b469a6ddf2af0354c43a39ded18b1aaf2a85c9449f094a6f"}
Sep 17 00:17:31.312902 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:31.312876    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-kdnhc" event={"ID":"0b40d7af-5ab8-4d80-9cf3-394b49de9e65","Type":"ContainerStarted","Data":"0ed4e1576c45655a82808e4cd95d22749db06539edb2c0c359a0d9b79b9ffea6"}
Sep 17 00:17:31.327367 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:31.327320    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vdppl" podStartSLOduration=2.260505555 podStartE2EDuration="5.327306178s" totalImagesPullingTime="3.066800623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:17:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:17:31.325689847 +0000 UTC m=+464.184182981" watchObservedRunningTime="2026-09-17 00:17:31.327306178 +0000 UTC m=+464.185799312"
Sep 17 00:17:31.339998 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:31.339951    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-kdnhc" podStartSLOduration=1.976014028 podStartE2EDuration="5.339940753s" totalImagesPullingTime="3.363926725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:17:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:17:31.338481138 +0000 UTC m=+464.196974267" watchObservedRunningTime="2026-09-17 00:17:31.339940753 +0000 UTC m=+464.198433891"
Sep 17 00:17:33.424430 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:33.424365    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j"]
Sep 17 00:17:33.443665 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:33.443639    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j"]
Sep 17 00:17:33.443813 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:33.443761    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j"
Sep 17 00:17:33.446293 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:33.446270    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 00:17:33.447280 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:33.447264    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 00:17:33.447365 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:33.447282    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-q8lnv\""
Sep 17 00:17:33.489770 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:33.489741    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b30c734-29df-445c-8668-f74c7adce423-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j\" (UID: \"6b30c734-29df-445c-8668-f74c7adce423\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j"
Sep 17 00:17:33.489932 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:33.489782    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b30c734-29df-445c-8668-f74c7adce423-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j\" (UID: \"6b30c734-29df-445c-8668-f74c7adce423\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j"
Sep 17 00:17:33.489932 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:33.489862    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2jgv\" (UniqueName: \"kubernetes.io/projected/6b30c734-29df-445c-8668-f74c7adce423-kube-api-access-n2jgv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j\" (UID: \"6b30c734-29df-445c-8668-f74c7adce423\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j"
Sep 17 00:17:33.591068 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:33.591034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b30c734-29df-445c-8668-f74c7adce423-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j\" (UID: \"6b30c734-29df-445c-8668-f74c7adce423\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j"
Sep 17 00:17:33.591068 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:33.591073    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b30c734-29df-445c-8668-f74c7adce423-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j\" (UID: \"6b30c734-29df-445c-8668-f74c7adce423\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j"
Sep 17 00:17:33.591305 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:33.591109    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2jgv\" (UniqueName: \"kubernetes.io/projected/6b30c734-29df-445c-8668-f74c7adce423-kube-api-access-n2jgv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j\" (UID: \"6b30c734-29df-445c-8668-f74c7adce423\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j"
Sep 17 00:17:33.591439 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:33.591390    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b30c734-29df-445c-8668-f74c7adce423-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j\" (UID: \"6b30c734-29df-445c-8668-f74c7adce423\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j"
Sep 17 00:17:33.591515 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:33.591488    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b30c734-29df-445c-8668-f74c7adce423-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j\" (UID: \"6b30c734-29df-445c-8668-f74c7adce423\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j"
Sep 17 00:17:33.598395 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:33.598373    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2jgv\" (UniqueName: \"kubernetes.io/projected/6b30c734-29df-445c-8668-f74c7adce423-kube-api-access-n2jgv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j\" (UID: \"6b30c734-29df-445c-8668-f74c7adce423\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j"
Sep 17 00:17:33.752856 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:33.752809    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j"
Sep 17 00:17:34.044389 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:34.044310    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j"]
Sep 17 00:17:34.047708 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:17:34.047680    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b30c734_29df_445c_8668_f74c7adce423.slice/crio-928d29c382bde2b76395591964d5d3591366514e402b3c77d51eead7ac1aba97 WatchSource:0}: Error finding container 928d29c382bde2b76395591964d5d3591366514e402b3c77d51eead7ac1aba97: Status 404 returned error can't find the container with id 928d29c382bde2b76395591964d5d3591366514e402b3c77d51eead7ac1aba97
Sep 17 00:17:34.326156 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:34.326072    2577 generic.go:356] "Generic (PLEG): container finished" podID="6b30c734-29df-445c-8668-f74c7adce423" containerID="a8349a439b7f9e35e2a418ab1449c1c737d9b7e2cb3da47c9c371a1a01ddd372" exitCode=0
Sep 17 00:17:34.326311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:34.326168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j" event={"ID":"6b30c734-29df-445c-8668-f74c7adce423","Type":"ContainerDied","Data":"a8349a439b7f9e35e2a418ab1449c1c737d9b7e2cb3da47c9c371a1a01ddd372"}
Sep 17 00:17:34.326311 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:34.326224    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j" event={"ID":"6b30c734-29df-445c-8668-f74c7adce423","Type":"ContainerStarted","Data":"928d29c382bde2b76395591964d5d3591366514e402b3c77d51eead7ac1aba97"}
Sep 17 00:17:35.332208 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:35.332129    2577 generic.go:356] "Generic (PLEG): container finished" podID="6b30c734-29df-445c-8668-f74c7adce423" containerID="c47960d864014bc4f1068d271ea51d4cbaf00549f953dc758584ec409573bc57" exitCode=0
Sep 17 00:17:35.332568 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:35.332210    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j" event={"ID":"6b30c734-29df-445c-8668-f74c7adce423","Type":"ContainerDied","Data":"c47960d864014bc4f1068d271ea51d4cbaf00549f953dc758584ec409573bc57"}
Sep 17 00:17:36.337394 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:36.337360    2577 generic.go:356] "Generic (PLEG): container finished" podID="6b30c734-29df-445c-8668-f74c7adce423" containerID="190a23f16f90f3aa3e707de482540ee1098b863d9d6fa16d2c7ba3350370a3d5" exitCode=0
Sep 17 00:17:36.337788 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:36.337408    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j" event={"ID":"6b30c734-29df-445c-8668-f74c7adce423","Type":"ContainerDied","Data":"190a23f16f90f3aa3e707de482540ee1098b863d9d6fa16d2c7ba3350370a3d5"}
Sep 17 00:17:37.453669 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:37.453645    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j"
Sep 17 00:17:37.518695 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:37.518665    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b30c734-29df-445c-8668-f74c7adce423-util\") pod \"6b30c734-29df-445c-8668-f74c7adce423\" (UID: \"6b30c734-29df-445c-8668-f74c7adce423\") "
Sep 17 00:17:37.518852 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:37.518716    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2jgv\" (UniqueName: \"kubernetes.io/projected/6b30c734-29df-445c-8668-f74c7adce423-kube-api-access-n2jgv\") pod \"6b30c734-29df-445c-8668-f74c7adce423\" (UID: \"6b30c734-29df-445c-8668-f74c7adce423\") "
Sep 17 00:17:37.518852 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:37.518738    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b30c734-29df-445c-8668-f74c7adce423-bundle\") pod \"6b30c734-29df-445c-8668-f74c7adce423\" (UID: \"6b30c734-29df-445c-8668-f74c7adce423\") "
Sep 17 00:17:37.519493 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:37.519467    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b30c734-29df-445c-8668-f74c7adce423-bundle" (OuterVolumeSpecName: "bundle") pod "6b30c734-29df-445c-8668-f74c7adce423" (UID: "6b30c734-29df-445c-8668-f74c7adce423"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:17:37.520729 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:37.520710    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b30c734-29df-445c-8668-f74c7adce423-kube-api-access-n2jgv" (OuterVolumeSpecName: "kube-api-access-n2jgv") pod "6b30c734-29df-445c-8668-f74c7adce423" (UID: "6b30c734-29df-445c-8668-f74c7adce423"). InnerVolumeSpecName "kube-api-access-n2jgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:17:37.526382 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:37.526358    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b30c734-29df-445c-8668-f74c7adce423-util" (OuterVolumeSpecName: "util") pod "6b30c734-29df-445c-8668-f74c7adce423" (UID: "6b30c734-29df-445c-8668-f74c7adce423"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:17:37.619254 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:37.619231    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b30c734-29df-445c-8668-f74c7adce423-util\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:17:37.619254 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:37.619253    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2jgv\" (UniqueName: \"kubernetes.io/projected/6b30c734-29df-445c-8668-f74c7adce423-kube-api-access-n2jgv\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:17:37.619388 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:37.619263    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b30c734-29df-445c-8668-f74c7adce423-bundle\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:17:38.346995 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:38.346962    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j" event={"ID":"6b30c734-29df-445c-8668-f74c7adce423","Type":"ContainerDied","Data":"928d29c382bde2b76395591964d5d3591366514e402b3c77d51eead7ac1aba97"}
Sep 17 00:17:38.347165 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:38.347010    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="928d29c382bde2b76395591964d5d3591366514e402b3c77d51eead7ac1aba97"
Sep 17 00:17:38.347165 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:38.347012    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcv59j"
Sep 17 00:17:47.636177 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.636149    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m"]
Sep 17 00:17:47.636610 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.636366    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b30c734-29df-445c-8668-f74c7adce423" containerName="pull"
Sep 17 00:17:47.636610 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.636377    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b30c734-29df-445c-8668-f74c7adce423" containerName="pull"
Sep 17 00:17:47.636610 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.636391    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b30c734-29df-445c-8668-f74c7adce423" containerName="extract"
Sep 17 00:17:47.636610 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.636396    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b30c734-29df-445c-8668-f74c7adce423" containerName="extract"
Sep 17 00:17:47.636610 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.636405    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b30c734-29df-445c-8668-f74c7adce423" containerName="util"
Sep 17 00:17:47.636610 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.636426    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b30c734-29df-445c-8668-f74c7adce423" containerName="util"
Sep 17 00:17:47.636610 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.636481    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b30c734-29df-445c-8668-f74c7adce423" containerName="extract"
Sep 17 00:17:47.638451 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.638410    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m"
Sep 17 00:17:47.641208 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.641187    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 17 00:17:47.641366 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.641189    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 17 00:17:47.641366 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.641289    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mk2l7\""
Sep 17 00:17:47.641531 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.641507    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 17 00:17:47.641579 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.641557    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 17 00:17:47.655971 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.655950    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m"]
Sep 17 00:17:47.689273 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.689249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ffb945e2-7e64-4c22-a080-97685816fd25-manifests\") pod \"opendatahub-operator-controller-manager-5cc689bf9d-h477m\" (UID: \"ffb945e2-7e64-4c22-a080-97685816fd25\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m"
Sep 17 00:17:47.689493 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.689278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ffb945e2-7e64-4c22-a080-97685816fd25-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cc689bf9d-h477m\" (UID: \"ffb945e2-7e64-4c22-a080-97685816fd25\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m"
Sep 17 00:17:47.689493 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.689299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ffb945e2-7e64-4c22-a080-97685816fd25-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cc689bf9d-h477m\" (UID: \"ffb945e2-7e64-4c22-a080-97685816fd25\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m"
Sep 17 00:17:47.689493 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.689324    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7576t\" (UniqueName: \"kubernetes.io/projected/ffb945e2-7e64-4c22-a080-97685816fd25-kube-api-access-7576t\") pod \"opendatahub-operator-controller-manager-5cc689bf9d-h477m\" (UID: \"ffb945e2-7e64-4c22-a080-97685816fd25\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m"
Sep 17 00:17:47.689493 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.689395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffb945e2-7e64-4c22-a080-97685816fd25-tmp\") pod \"opendatahub-operator-controller-manager-5cc689bf9d-h477m\" (UID: \"ffb945e2-7e64-4c22-a080-97685816fd25\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m"
Sep 17 00:17:47.790640 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.790604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7576t\" (UniqueName: \"kubernetes.io/projected/ffb945e2-7e64-4c22-a080-97685816fd25-kube-api-access-7576t\") pod \"opendatahub-operator-controller-manager-5cc689bf9d-h477m\" (UID: \"ffb945e2-7e64-4c22-a080-97685816fd25\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m"
Sep 17 00:17:47.790806 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.790656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffb945e2-7e64-4c22-a080-97685816fd25-tmp\") pod \"opendatahub-operator-controller-manager-5cc689bf9d-h477m\" (UID: \"ffb945e2-7e64-4c22-a080-97685816fd25\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m"
Sep 17 00:17:47.790806 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.790692    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ffb945e2-7e64-4c22-a080-97685816fd25-manifests\") pod \"opendatahub-operator-controller-manager-5cc689bf9d-h477m\" (UID: \"ffb945e2-7e64-4c22-a080-97685816fd25\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m"
Sep 17 00:17:47.790806 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.790708    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ffb945e2-7e64-4c22-a080-97685816fd25-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cc689bf9d-h477m\" (UID: \"ffb945e2-7e64-4c22-a080-97685816fd25\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m"
Sep 17 00:17:47.790806 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.790726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ffb945e2-7e64-4c22-a080-97685816fd25-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cc689bf9d-h477m\" (UID: \"ffb945e2-7e64-4c22-a080-97685816fd25\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m"
Sep 17 00:17:47.791116 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.791091    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ffb945e2-7e64-4c22-a080-97685816fd25-manifests\") pod \"opendatahub-operator-controller-manager-5cc689bf9d-h477m\" (UID: \"ffb945e2-7e64-4c22-a080-97685816fd25\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m"
Sep 17 00:17:47.792996 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.792964    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffb945e2-7e64-4c22-a080-97685816fd25-tmp\") pod \"opendatahub-operator-controller-manager-5cc689bf9d-h477m\" (UID: \"ffb945e2-7e64-4c22-a080-97685816fd25\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m"
Sep 17 00:17:47.793201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.793176    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ffb945e2-7e64-4c22-a080-97685816fd25-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cc689bf9d-h477m\" (UID: \"ffb945e2-7e64-4c22-a080-97685816fd25\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m"
Sep 17 00:17:47.793305 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.793287    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ffb945e2-7e64-4c22-a080-97685816fd25-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cc689bf9d-h477m\" (UID: \"ffb945e2-7e64-4c22-a080-97685816fd25\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m"
Sep 17 00:17:47.798141 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.798117    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7576t\" (UniqueName: \"kubernetes.io/projected/ffb945e2-7e64-4c22-a080-97685816fd25-kube-api-access-7576t\") pod \"opendatahub-operator-controller-manager-5cc689bf9d-h477m\" (UID: \"ffb945e2-7e64-4c22-a080-97685816fd25\") " pod="opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m"
Sep 17 00:17:47.838693 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.838664    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj"]
Sep 17 00:17:47.840960 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.840945    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj"
Sep 17 00:17:47.843388 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.843371    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 00:17:47.843510 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.843371    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 00:17:47.843510 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.843408    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-q8lnv\""
Sep 17 00:17:47.849624 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.849599    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj"]
Sep 17 00:17:47.891313 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.891252    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5zms\" (UniqueName: \"kubernetes.io/projected/7ee23551-1fcc-42fe-b3d9-d12d3ee4b691-kube-api-access-s5zms\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj\" (UID: \"7ee23551-1fcc-42fe-b3d9-d12d3ee4b691\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj"
Sep 17 00:17:47.891313 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.891301    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ee23551-1fcc-42fe-b3d9-d12d3ee4b691-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj\" (UID: \"7ee23551-1fcc-42fe-b3d9-d12d3ee4b691\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj"
Sep 17 00:17:47.891499 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.891342    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ee23551-1fcc-42fe-b3d9-d12d3ee4b691-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj\" (UID: \"7ee23551-1fcc-42fe-b3d9-d12d3ee4b691\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj"
Sep 17 00:17:47.948491 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.948462    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m"
Sep 17 00:17:47.992204 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.992166    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ee23551-1fcc-42fe-b3d9-d12d3ee4b691-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj\" (UID: \"7ee23551-1fcc-42fe-b3d9-d12d3ee4b691\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj"
Sep 17 00:17:47.992357 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.992223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ee23551-1fcc-42fe-b3d9-d12d3ee4b691-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj\" (UID: \"7ee23551-1fcc-42fe-b3d9-d12d3ee4b691\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj"
Sep 17 00:17:47.992357 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.992305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5zms\" (UniqueName: \"kubernetes.io/projected/7ee23551-1fcc-42fe-b3d9-d12d3ee4b691-kube-api-access-s5zms\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj\" (UID: \"7ee23551-1fcc-42fe-b3d9-d12d3ee4b691\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj"
Sep 17 00:17:47.992676 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.992640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ee23551-1fcc-42fe-b3d9-d12d3ee4b691-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj\" (UID: \"7ee23551-1fcc-42fe-b3d9-d12d3ee4b691\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj"
Sep 17 00:17:47.992778 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:47.992676    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ee23551-1fcc-42fe-b3d9-d12d3ee4b691-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj\" (UID: \"7ee23551-1fcc-42fe-b3d9-d12d3ee4b691\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj"
Sep 17 00:17:48.011353 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:48.011099    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5zms\" (UniqueName: \"kubernetes.io/projected/7ee23551-1fcc-42fe-b3d9-d12d3ee4b691-kube-api-access-s5zms\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj\" (UID: \"7ee23551-1fcc-42fe-b3d9-d12d3ee4b691\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj"
Sep 17 00:17:48.150604 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:48.150516    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj"
Sep 17 00:17:48.222654 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:48.222630    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m"]
Sep 17 00:17:48.225180 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:17:48.225147    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb945e2_7e64_4c22_a080_97685816fd25.slice/crio-26095c134756f3842586ef5e6feba379bfb89ac74433e301221a7f5acd17d1bd WatchSource:0}: Error finding container 26095c134756f3842586ef5e6feba379bfb89ac74433e301221a7f5acd17d1bd: Status 404 returned error can't find the container with id 26095c134756f3842586ef5e6feba379bfb89ac74433e301221a7f5acd17d1bd
Sep 17 00:17:48.382586 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:48.382549    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m" event={"ID":"ffb945e2-7e64-4c22-a080-97685816fd25","Type":"ContainerStarted","Data":"26095c134756f3842586ef5e6feba379bfb89ac74433e301221a7f5acd17d1bd"}
Sep 17 00:17:48.423195 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:48.423118    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj"]
Sep 17 00:17:48.425726 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:17:48.425699    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee23551_1fcc_42fe_b3d9_d12d3ee4b691.slice/crio-71c65b14846ed4810c189626709b94bea40760270bcb5f89dfa1d521eb5bb30c WatchSource:0}: Error finding container 71c65b14846ed4810c189626709b94bea40760270bcb5f89dfa1d521eb5bb30c: Status 404 returned error can't find the container with id 71c65b14846ed4810c189626709b94bea40760270bcb5f89dfa1d521eb5bb30c
Sep 17 00:17:49.393936 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:49.393901    2577 generic.go:356] "Generic (PLEG): container finished" podID="7ee23551-1fcc-42fe-b3d9-d12d3ee4b691" containerID="962a1ebc13bfcced18287827e55e1f4f89e2e3d20b39daa2d91e95561bf0f804" exitCode=0
Sep 17 00:17:49.394388 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:49.393941    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj" event={"ID":"7ee23551-1fcc-42fe-b3d9-d12d3ee4b691","Type":"ContainerDied","Data":"962a1ebc13bfcced18287827e55e1f4f89e2e3d20b39daa2d91e95561bf0f804"}
Sep 17 00:17:49.394388 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:49.393998    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj" event={"ID":"7ee23551-1fcc-42fe-b3d9-d12d3ee4b691","Type":"ContainerStarted","Data":"71c65b14846ed4810c189626709b94bea40760270bcb5f89dfa1d521eb5bb30c"}
Sep 17 00:17:52.409437 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:52.409388    2577 generic.go:356] "Generic (PLEG): container finished" podID="7ee23551-1fcc-42fe-b3d9-d12d3ee4b691" containerID="c7434da8e89bf146c39afb4275b9cd474eeed482ace4ab5bb2eb159685f49039" exitCode=0
Sep 17 00:17:52.409873 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:52.409460    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj" event={"ID":"7ee23551-1fcc-42fe-b3d9-d12d3ee4b691","Type":"ContainerDied","Data":"c7434da8e89bf146c39afb4275b9cd474eeed482ace4ab5bb2eb159685f49039"}
Sep 17 00:17:52.411199 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:52.411178    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m" event={"ID":"ffb945e2-7e64-4c22-a080-97685816fd25","Type":"ContainerStarted","Data":"b12d175a075b9ecd781138ae09e8b570f23677b066ea8223b64af3cc3bab78a5"}
Sep 17 00:17:53.416829 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:53.416796    2577 generic.go:356] "Generic (PLEG): container finished" podID="7ee23551-1fcc-42fe-b3d9-d12d3ee4b691" containerID="789dc243ff7406150e7875913bb13def80b109ff8eef03e081ad647c1e5b3a15" exitCode=0
Sep 17 00:17:53.417205 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:53.416887    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj" event={"ID":"7ee23551-1fcc-42fe-b3d9-d12d3ee4b691","Type":"ContainerDied","Data":"789dc243ff7406150e7875913bb13def80b109ff8eef03e081ad647c1e5b3a15"}
Sep 17 00:17:53.418298 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:53.418274    2577 generic.go:356] "Generic (PLEG): container finished" podID="ffb945e2-7e64-4c22-a080-97685816fd25" containerID="b12d175a075b9ecd781138ae09e8b570f23677b066ea8223b64af3cc3bab78a5" exitCode=0
Sep 17 00:17:53.418405 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:53.418336    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m" event={"ID":"ffb945e2-7e64-4c22-a080-97685816fd25","Type":"ContainerDied","Data":"b12d175a075b9ecd781138ae09e8b570f23677b066ea8223b64af3cc3bab78a5"}
Sep 17 00:17:54.424367 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:54.424329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m" event={"ID":"ffb945e2-7e64-4c22-a080-97685816fd25","Type":"ContainerStarted","Data":"3a1d401f414f138d468274640305734e23f849bfdab0c0f7558198d594050724"}
Sep 17 00:17:54.424857 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:54.424445    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m"
Sep 17 00:17:54.444243 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:54.444192    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m" podStartSLOduration=2.626352052 podStartE2EDuration="7.444175604s" totalImagesPullingTime="3.754862444s" totalInitContainerRuntime="1.062961108s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:17:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:17:54.443644933 +0000 UTC m=+487.302138095" watchObservedRunningTime="2026-09-17 00:17:54.444175604 +0000 UTC m=+487.302668739"
Sep 17 00:17:54.547653 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:54.547624    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj"
Sep 17 00:17:54.646565 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:54.646529    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5zms\" (UniqueName: \"kubernetes.io/projected/7ee23551-1fcc-42fe-b3d9-d12d3ee4b691-kube-api-access-s5zms\") pod \"7ee23551-1fcc-42fe-b3d9-d12d3ee4b691\" (UID: \"7ee23551-1fcc-42fe-b3d9-d12d3ee4b691\") "
Sep 17 00:17:54.646565 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:54.646565    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ee23551-1fcc-42fe-b3d9-d12d3ee4b691-util\") pod \"7ee23551-1fcc-42fe-b3d9-d12d3ee4b691\" (UID: \"7ee23551-1fcc-42fe-b3d9-d12d3ee4b691\") "
Sep 17 00:17:54.646781 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:54.646592    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ee23551-1fcc-42fe-b3d9-d12d3ee4b691-bundle\") pod \"7ee23551-1fcc-42fe-b3d9-d12d3ee4b691\" (UID: \"7ee23551-1fcc-42fe-b3d9-d12d3ee4b691\") "
Sep 17 00:17:54.647286 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:54.647253    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee23551-1fcc-42fe-b3d9-d12d3ee4b691-bundle" (OuterVolumeSpecName: "bundle") pod "7ee23551-1fcc-42fe-b3d9-d12d3ee4b691" (UID: "7ee23551-1fcc-42fe-b3d9-d12d3ee4b691"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:17:54.648589 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:54.648560    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee23551-1fcc-42fe-b3d9-d12d3ee4b691-kube-api-access-s5zms" (OuterVolumeSpecName: "kube-api-access-s5zms") pod "7ee23551-1fcc-42fe-b3d9-d12d3ee4b691" (UID: "7ee23551-1fcc-42fe-b3d9-d12d3ee4b691"). InnerVolumeSpecName "kube-api-access-s5zms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:17:54.651578 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:54.651547    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee23551-1fcc-42fe-b3d9-d12d3ee4b691-util" (OuterVolumeSpecName: "util") pod "7ee23551-1fcc-42fe-b3d9-d12d3ee4b691" (UID: "7ee23551-1fcc-42fe-b3d9-d12d3ee4b691"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:17:54.747801 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:54.747773    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5zms\" (UniqueName: \"kubernetes.io/projected/7ee23551-1fcc-42fe-b3d9-d12d3ee4b691-kube-api-access-s5zms\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:17:54.747801 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:54.747799    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ee23551-1fcc-42fe-b3d9-d12d3ee4b691-util\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:17:54.747957 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:54.747809    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ee23551-1fcc-42fe-b3d9-d12d3ee4b691-bundle\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:17:55.430355 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:55.430330    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj"
Sep 17 00:17:55.430759 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:55.430364    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0cjsdj" event={"ID":"7ee23551-1fcc-42fe-b3d9-d12d3ee4b691","Type":"ContainerDied","Data":"71c65b14846ed4810c189626709b94bea40760270bcb5f89dfa1d521eb5bb30c"}
Sep 17 00:17:55.430759 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:17:55.430426    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71c65b14846ed4810c189626709b94bea40760270bcb5f89dfa1d521eb5bb30c"
Sep 17 00:18:05.433051 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:05.433016    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5cc689bf9d-h477m"
Sep 17 00:18:07.149570 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:07.149533    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5f57c6d865-67rlh"]
Sep 17 00:18:07.149936 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:07.149772    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee23551-1fcc-42fe-b3d9-d12d3ee4b691" containerName="extract"
Sep 17 00:18:07.149936 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:07.149782    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee23551-1fcc-42fe-b3d9-d12d3ee4b691" containerName="extract"
Sep 17 00:18:07.149936 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:07.149791    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee23551-1fcc-42fe-b3d9-d12d3ee4b691" containerName="pull"
Sep 17 00:18:07.149936 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:07.149797    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee23551-1fcc-42fe-b3d9-d12d3ee4b691" containerName="pull"
Sep 17 00:18:07.149936 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:07.149812    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee23551-1fcc-42fe-b3d9-d12d3ee4b691" containerName="util"
Sep 17 00:18:07.149936 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:07.149817    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee23551-1fcc-42fe-b3d9-d12d3ee4b691" containerName="util"
Sep 17 00:18:07.149936 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:07.149870    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee23551-1fcc-42fe-b3d9-d12d3ee4b691" containerName="extract"
Sep 17 00:18:07.154457 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:07.154438    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f57c6d865-67rlh"
Sep 17 00:18:07.157849 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:07.157822    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 17 00:18:07.157995 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:07.157853    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 00:18:07.157995 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:07.157872    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 17 00:18:07.159528 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:07.159510    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 00:18:07.172344 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:07.172315    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f57c6d865-67rlh"]
Sep 17 00:18:07.236875 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:07.236839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8305f43c-0a13-42e3-89f3-67da95cbd3bd-tls-certs\") pod \"kube-auth-proxy-5f57c6d865-67rlh\" (UID: \"8305f43c-0a13-42e3-89f3-67da95cbd3bd\") " pod="openshift-ingress/kube-auth-proxy-5f57c6d865-67rlh"
Sep 17 00:18:07.237030 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:07.236888    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8305f43c-0a13-42e3-89f3-67da95cbd3bd-tmp\") pod \"kube-auth-proxy-5f57c6d865-67rlh\" (UID: \"8305f43c-0a13-42e3-89f3-67da95cbd3bd\") " pod="openshift-ingress/kube-auth-proxy-5f57c6d865-67rlh"
Sep 17 00:18:07.237030 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:07.236978    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxfvz\" (UniqueName: \"kubernetes.io/projected/8305f43c-0a13-42e3-89f3-67da95cbd3bd-kube-api-access-cxfvz\") pod \"kube-auth-proxy-5f57c6d865-67rlh\" (UID: \"8305f43c-0a13-42e3-89f3-67da95cbd3bd\") " pod="openshift-ingress/kube-auth-proxy-5f57c6d865-67rlh"
Sep 17 00:18:07.337767 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:07.337719    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8305f43c-0a13-42e3-89f3-67da95cbd3bd-tmp\") pod \"kube-auth-proxy-5f57c6d865-67rlh\" (UID: \"8305f43c-0a13-42e3-89f3-67da95cbd3bd\") " pod="openshift-ingress/kube-auth-proxy-5f57c6d865-67rlh"
Sep 17 00:18:07.337945 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:07.337791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxfvz\" (UniqueName: \"kubernetes.io/projected/8305f43c-0a13-42e3-89f3-67da95cbd3bd-kube-api-access-cxfvz\") pod \"kube-auth-proxy-5f57c6d865-67rlh\" (UID: \"8305f43c-0a13-42e3-89f3-67da95cbd3bd\") " pod="openshift-ingress/kube-auth-proxy-5f57c6d865-67rlh"
Sep 17 00:18:07.337945 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:07.337819    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8305f43c-0a13-42e3-89f3-67da95cbd3bd-tls-certs\") pod \"kube-auth-proxy-5f57c6d865-67rlh\" (UID: \"8305f43c-0a13-42e3-89f3-67da95cbd3bd\") " pod="openshift-ingress/kube-auth-proxy-5f57c6d865-67rlh"
Sep 17 00:18:07.337945 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:18:07.337895    2577 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 17 00:18:07.337945 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:18:07.337944    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8305f43c-0a13-42e3-89f3-67da95cbd3bd-tls-certs podName:8305f43c-0a13-42e3-89f3-67da95cbd3bd nodeName:}" failed. No retries permitted until 2026-09-17 00:18:07.837930161 +0000 UTC m=+500.696423274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/8305f43c-0a13-42e3-89f3-67da95cbd3bd-tls-certs") pod "kube-auth-proxy-5f57c6d865-67rlh" (UID: "8305f43c-0a13-42e3-89f3-67da95cbd3bd") : secret "kube-auth-proxy-tls" not found
Sep 17 00:18:07.339973 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:07.339950    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8305f43c-0a13-42e3-89f3-67da95cbd3bd-tmp\") pod \"kube-auth-proxy-5f57c6d865-67rlh\" (UID: \"8305f43c-0a13-42e3-89f3-67da95cbd3bd\") " pod="openshift-ingress/kube-auth-proxy-5f57c6d865-67rlh"
Sep 17 00:18:07.349410 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:07.349383    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxfvz\" (UniqueName: \"kubernetes.io/projected/8305f43c-0a13-42e3-89f3-67da95cbd3bd-kube-api-access-cxfvz\") pod \"kube-auth-proxy-5f57c6d865-67rlh\" (UID: \"8305f43c-0a13-42e3-89f3-67da95cbd3bd\") " pod="openshift-ingress/kube-auth-proxy-5f57c6d865-67rlh"
Sep 17 00:18:07.843038 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:07.842997    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8305f43c-0a13-42e3-89f3-67da95cbd3bd-tls-certs\") pod \"kube-auth-proxy-5f57c6d865-67rlh\" (UID: \"8305f43c-0a13-42e3-89f3-67da95cbd3bd\") " pod="openshift-ingress/kube-auth-proxy-5f57c6d865-67rlh"
Sep 17 00:18:07.845532 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:07.845508    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8305f43c-0a13-42e3-89f3-67da95cbd3bd-tls-certs\") pod \"kube-auth-proxy-5f57c6d865-67rlh\" (UID: \"8305f43c-0a13-42e3-89f3-67da95cbd3bd\") " pod="openshift-ingress/kube-auth-proxy-5f57c6d865-67rlh"
Sep 17 00:18:08.063070 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.063030    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f57c6d865-67rlh"
Sep 17 00:18:08.334597 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.334462    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f57c6d865-67rlh"]
Sep 17 00:18:08.336631 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:18:08.336607    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8305f43c_0a13_42e3_89f3_67da95cbd3bd.slice/crio-ce9bb90f23757cf618de3832ea3979e2d1d2b92dd3be43b7112f4e3c2200968c WatchSource:0}: Error finding container ce9bb90f23757cf618de3832ea3979e2d1d2b92dd3be43b7112f4e3c2200968c: Status 404 returned error can't find the container with id ce9bb90f23757cf618de3832ea3979e2d1d2b92dd3be43b7112f4e3c2200968c
Sep 17 00:18:08.405297 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.405217    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"]
Sep 17 00:18:08.408061 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.408045    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:08.410592 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.410571    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 17 00:18:08.410827 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.410625    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 17 00:18:08.410827 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.410649    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 17 00:18:08.411131 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.410884    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-88s5c\""
Sep 17 00:18:08.411131 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.410955    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 00:18:08.421800 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.421774    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"]
Sep 17 00:18:08.483184 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.483150    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f57c6d865-67rlh" event={"ID":"8305f43c-0a13-42e3-89f3-67da95cbd3bd","Type":"ContainerStarted","Data":"ce9bb90f23757cf618de3832ea3979e2d1d2b92dd3be43b7112f4e3c2200968c"}
Sep 17 00:18:08.548985 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.548940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h66p\" (UniqueName: \"kubernetes.io/projected/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-kube-api-access-2h66p\") pod \"istiod-openshift-gateway-7b7995d5cd-lqnps\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:08.548985 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.548984    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lqnps\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:08.549214 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.549001    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lqnps\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:08.549214 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.549019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lqnps\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:08.549214 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.549061    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lqnps\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:08.549214 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.549120    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lqnps\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:08.549214 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.549151    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lqnps\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:08.649744 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.649693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h66p\" (UniqueName: \"kubernetes.io/projected/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-kube-api-access-2h66p\") pod \"istiod-openshift-gateway-7b7995d5cd-lqnps\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:08.649990 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.649749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lqnps\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:08.649990 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.649887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lqnps\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:08.649990 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.649945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lqnps\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:08.649990 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.649970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lqnps\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:08.650204 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.650024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lqnps\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:08.650204 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.650055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lqnps\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:08.650811 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.650777    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lqnps\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:08.652407 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.652385    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lqnps\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:08.652407 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.652400    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lqnps\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:08.652626 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.652610    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lqnps\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:08.652739 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.652716    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lqnps\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:08.657551 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.657496    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h66p\" (UniqueName: \"kubernetes.io/projected/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-kube-api-access-2h66p\") pod \"istiod-openshift-gateway-7b7995d5cd-lqnps\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:08.657638 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.657624    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lqnps\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:08.717665 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.717627    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:08.990062 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:08.989979    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"]
Sep 17 00:18:08.992470 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:18:08.992439    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33b9f0c_8f41_48e4_a6a3_a6e7b2b70490.slice/crio-2287e9dc9bd4c5374028fdb2f9ecb17bee774633f8f25f2aa30ff0bf4ea241b0 WatchSource:0}: Error finding container 2287e9dc9bd4c5374028fdb2f9ecb17bee774633f8f25f2aa30ff0bf4ea241b0: Status 404 returned error can't find the container with id 2287e9dc9bd4c5374028fdb2f9ecb17bee774633f8f25f2aa30ff0bf4ea241b0
Sep 17 00:18:09.489485 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:09.489436    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps" event={"ID":"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490","Type":"ContainerStarted","Data":"2287e9dc9bd4c5374028fdb2f9ecb17bee774633f8f25f2aa30ff0bf4ea241b0"}
Sep 17 00:18:12.505458 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:12.505404    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f57c6d865-67rlh" event={"ID":"8305f43c-0a13-42e3-89f3-67da95cbd3bd","Type":"ContainerStarted","Data":"7ff5b72ce088592e80be4d91582507a4691cddaad3d71ccade036d2208e433b3"}
Sep 17 00:18:12.522681 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:12.522627    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5f57c6d865-67rlh" podStartSLOduration=2.368000005 podStartE2EDuration="5.522605977s" totalImagesPullingTime="3.154605972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:18:12.522168566 +0000 UTC m=+505.380661700" watchObservedRunningTime="2026-09-17 00:18:12.522605977 +0000 UTC m=+505.381099113"
Sep 17 00:18:13.580974 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:13.580938    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 00:18:13.581194 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:13.581008    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 00:18:14.515927 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:14.515892    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps" event={"ID":"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490","Type":"ContainerStarted","Data":"3977268be5d759d46989aedac60bf107f8e022c4a0e0105366728e54c994d88a"}
Sep 17 00:18:14.516099 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:14.516057    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:14.517459 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:14.517430    2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-lqnps container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 00:18:14.517565 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:14.517478    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps" podUID="b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 00:18:14.564683 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:14.564635    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps" podStartSLOduration=1.978834291 podStartE2EDuration="6.564620743s" totalImagesPullingTime="4.585786452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:18:14.562170377 +0000 UTC m=+507.420663512" watchObservedRunningTime="2026-09-17 00:18:14.564620743 +0000 UTC m=+507.423113878"
Sep 17 00:18:15.521189 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:15.521156    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:28.325372 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:28.325330    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mnjsh"]
Sep 17 00:18:28.332307 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:28.332279    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mnjsh"
Sep 17 00:18:28.336536 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:28.336512    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-8f66d\""
Sep 17 00:18:28.336668 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:28.336595    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 17 00:18:28.347203 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:28.347172    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-96l8l"]
Sep 17 00:18:28.349239 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:28.349216    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mnjsh"]
Sep 17 00:18:28.349350 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:28.349303    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-96l8l"
Sep 17 00:18:28.353566 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:28.353534    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-q78jx\""
Sep 17 00:18:28.353725 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:28.353708    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 17 00:18:28.374107 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:28.374071    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-96l8l"]
Sep 17 00:18:28.407643 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:28.407604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e37901a4-81e2-41bb-bb2a-1acf51d61759-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-96l8l\" (UID: \"e37901a4-81e2-41bb-bb2a-1acf51d61759\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-96l8l"
Sep 17 00:18:28.407643 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:28.407650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7npsf\" (UniqueName: \"kubernetes.io/projected/e37901a4-81e2-41bb-bb2a-1acf51d61759-kube-api-access-7npsf\") pod \"llmisvc-controller-manager-777c6bd7d5-96l8l\" (UID: \"e37901a4-81e2-41bb-bb2a-1acf51d61759\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-96l8l"
Sep 17 00:18:28.407886 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:28.407675    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkv5q\" (UniqueName: \"kubernetes.io/projected/c85fdce2-ade0-4b20-b82d-0decac04bf20-kube-api-access-nkv5q\") pod \"kserve-controller-manager-d6f4b65b8-mnjsh\" (UID: \"c85fdce2-ade0-4b20-b82d-0decac04bf20\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mnjsh"
Sep 17 00:18:28.407886 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:28.407708    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c85fdce2-ade0-4b20-b82d-0decac04bf20-cert\") pod \"kserve-controller-manager-d6f4b65b8-mnjsh\" (UID: \"c85fdce2-ade0-4b20-b82d-0decac04bf20\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mnjsh"
Sep 17 00:18:28.508803 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:28.508768    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c85fdce2-ade0-4b20-b82d-0decac04bf20-cert\") pod \"kserve-controller-manager-d6f4b65b8-mnjsh\" (UID: \"c85fdce2-ade0-4b20-b82d-0decac04bf20\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mnjsh"
Sep 17 00:18:28.509017 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:28.508818    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e37901a4-81e2-41bb-bb2a-1acf51d61759-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-96l8l\" (UID: \"e37901a4-81e2-41bb-bb2a-1acf51d61759\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-96l8l"
Sep 17 00:18:28.509017 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:28.508842    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7npsf\" (UniqueName: \"kubernetes.io/projected/e37901a4-81e2-41bb-bb2a-1acf51d61759-kube-api-access-7npsf\") pod \"llmisvc-controller-manager-777c6bd7d5-96l8l\" (UID: \"e37901a4-81e2-41bb-bb2a-1acf51d61759\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-96l8l"
Sep 17 00:18:28.509017 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:18:28.508932    2577 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 17 00:18:28.509017 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:28.508963    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkv5q\" (UniqueName: \"kubernetes.io/projected/c85fdce2-ade0-4b20-b82d-0decac04bf20-kube-api-access-nkv5q\") pod \"kserve-controller-manager-d6f4b65b8-mnjsh\" (UID: \"c85fdce2-ade0-4b20-b82d-0decac04bf20\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mnjsh"
Sep 17 00:18:28.509017 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:18:28.509004    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c85fdce2-ade0-4b20-b82d-0decac04bf20-cert podName:c85fdce2-ade0-4b20-b82d-0decac04bf20 nodeName:}" failed. No retries permitted until 2026-09-17 00:18:29.008984738 +0000 UTC m=+521.867477850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c85fdce2-ade0-4b20-b82d-0decac04bf20-cert") pod "kserve-controller-manager-d6f4b65b8-mnjsh" (UID: "c85fdce2-ade0-4b20-b82d-0decac04bf20") : secret "kserve-webhook-server-cert" not found
Sep 17 00:18:28.509017 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:18:28.508931    2577 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 17 00:18:28.509353 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:18:28.509094    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e37901a4-81e2-41bb-bb2a-1acf51d61759-cert podName:e37901a4-81e2-41bb-bb2a-1acf51d61759 nodeName:}" failed. No retries permitted until 2026-09-17 00:18:29.009075778 +0000 UTC m=+521.867568910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e37901a4-81e2-41bb-bb2a-1acf51d61759-cert") pod "llmisvc-controller-manager-777c6bd7d5-96l8l" (UID: "e37901a4-81e2-41bb-bb2a-1acf51d61759") : secret "llmisvc-webhook-server-cert" not found
Sep 17 00:18:28.519099 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:28.519068    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7npsf\" (UniqueName: \"kubernetes.io/projected/e37901a4-81e2-41bb-bb2a-1acf51d61759-kube-api-access-7npsf\") pod \"llmisvc-controller-manager-777c6bd7d5-96l8l\" (UID: \"e37901a4-81e2-41bb-bb2a-1acf51d61759\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-96l8l"
Sep 17 00:18:28.519243 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:28.519219    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkv5q\" (UniqueName: \"kubernetes.io/projected/c85fdce2-ade0-4b20-b82d-0decac04bf20-kube-api-access-nkv5q\") pod \"kserve-controller-manager-d6f4b65b8-mnjsh\" (UID: \"c85fdce2-ade0-4b20-b82d-0decac04bf20\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mnjsh"
Sep 17 00:18:29.013269 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:29.013222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c85fdce2-ade0-4b20-b82d-0decac04bf20-cert\") pod \"kserve-controller-manager-d6f4b65b8-mnjsh\" (UID: \"c85fdce2-ade0-4b20-b82d-0decac04bf20\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mnjsh"
Sep 17 00:18:29.013269 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:29.013279    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e37901a4-81e2-41bb-bb2a-1acf51d61759-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-96l8l\" (UID: \"e37901a4-81e2-41bb-bb2a-1acf51d61759\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-96l8l"
Sep 17 00:18:29.013583 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:18:29.013394    2577 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 17 00:18:29.013583 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:18:29.013483    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e37901a4-81e2-41bb-bb2a-1acf51d61759-cert podName:e37901a4-81e2-41bb-bb2a-1acf51d61759 nodeName:}" failed. No retries permitted until 2026-09-17 00:18:30.013468509 +0000 UTC m=+522.871961621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e37901a4-81e2-41bb-bb2a-1acf51d61759-cert") pod "llmisvc-controller-manager-777c6bd7d5-96l8l" (UID: "e37901a4-81e2-41bb-bb2a-1acf51d61759") : secret "llmisvc-webhook-server-cert" not found
Sep 17 00:18:29.015909 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:29.015882    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c85fdce2-ade0-4b20-b82d-0decac04bf20-cert\") pod \"kserve-controller-manager-d6f4b65b8-mnjsh\" (UID: \"c85fdce2-ade0-4b20-b82d-0decac04bf20\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mnjsh"
Sep 17 00:18:29.242175 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:29.242134    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mnjsh"
Sep 17 00:18:29.519540 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:29.519460    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mnjsh"]
Sep 17 00:18:29.522187 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:18:29.522159    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85fdce2_ade0_4b20_b82d_0decac04bf20.slice/crio-d61698ea73642353f2d162c3197405188b81a0a76b0839cdcc0ea32a04f0e9aa WatchSource:0}: Error finding container d61698ea73642353f2d162c3197405188b81a0a76b0839cdcc0ea32a04f0e9aa: Status 404 returned error can't find the container with id d61698ea73642353f2d162c3197405188b81a0a76b0839cdcc0ea32a04f0e9aa
Sep 17 00:18:29.582011 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:29.581925    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mnjsh" event={"ID":"c85fdce2-ade0-4b20-b82d-0decac04bf20","Type":"ContainerStarted","Data":"d61698ea73642353f2d162c3197405188b81a0a76b0839cdcc0ea32a04f0e9aa"}
Sep 17 00:18:30.020226 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:30.020186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e37901a4-81e2-41bb-bb2a-1acf51d61759-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-96l8l\" (UID: \"e37901a4-81e2-41bb-bb2a-1acf51d61759\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-96l8l"
Sep 17 00:18:30.022665 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:30.022632    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e37901a4-81e2-41bb-bb2a-1acf51d61759-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-96l8l\" (UID: \"e37901a4-81e2-41bb-bb2a-1acf51d61759\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-96l8l"
Sep 17 00:18:30.159037 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:30.158997    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-96l8l"
Sep 17 00:18:30.427343 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:30.427298    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-96l8l"]
Sep 17 00:18:30.429878 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:18:30.429848    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode37901a4_81e2_41bb_bb2a_1acf51d61759.slice/crio-7d33bb766c2932fd8482d4332d65121444d2e556c89434e7fec5e5ba7b0590a3 WatchSource:0}: Error finding container 7d33bb766c2932fd8482d4332d65121444d2e556c89434e7fec5e5ba7b0590a3: Status 404 returned error can't find the container with id 7d33bb766c2932fd8482d4332d65121444d2e556c89434e7fec5e5ba7b0590a3
Sep 17 00:18:30.589060 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:30.589005    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-96l8l" event={"ID":"e37901a4-81e2-41bb-bb2a-1acf51d61759","Type":"ContainerStarted","Data":"7d33bb766c2932fd8482d4332d65121444d2e556c89434e7fec5e5ba7b0590a3"}
Sep 17 00:18:30.969883 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:30.969838    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"]
Sep 17 00:18:30.970097 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:30.970072    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps" podUID="b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490" containerName="discovery" containerID="cri-o://3977268be5d759d46989aedac60bf107f8e022c4a0e0105366728e54c994d88a" gracePeriod=30
Sep 17 00:18:31.275081 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.275054    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:31.332302 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.332266    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-istio-csr-dns-cert\") pod \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") "
Sep 17 00:18:31.332513 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.332334    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-istio-token\") pod \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") "
Sep 17 00:18:31.332513 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.332482    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-local-certs\") pod \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") "
Sep 17 00:18:31.332645 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.332564    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-istio-csr-ca-configmap\") pod \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") "
Sep 17 00:18:31.332645 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.332621    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-istio-kubeconfig\") pod \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") "
Sep 17 00:18:31.332757 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.332655    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-cacerts\") pod \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") "
Sep 17 00:18:31.332757 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.332684    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h66p\" (UniqueName: \"kubernetes.io/projected/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-kube-api-access-2h66p\") pod \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\" (UID: \"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490\") "
Sep 17 00:18:31.333196 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.333168    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490" (UID: "b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:18:31.335441 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.335359    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-local-certs" (OuterVolumeSpecName: "local-certs") pod "b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490" (UID: "b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:18:31.335623 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.335603    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490" (UID: "b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:18:31.335710 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.335684    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-kube-api-access-2h66p" (OuterVolumeSpecName: "kube-api-access-2h66p") pod "b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490" (UID: "b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490"). InnerVolumeSpecName "kube-api-access-2h66p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:18:31.335913 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.335885    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-istio-token" (OuterVolumeSpecName: "istio-token") pod "b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490" (UID: "b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:18:31.336010 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.335911    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490" (UID: "b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:18:31.336010 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.335982    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-cacerts" (OuterVolumeSpecName: "cacerts") pod "b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490" (UID: "b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:18:31.433529 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.433493    2577 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-local-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:18:31.433529 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.433531    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-istio-csr-ca-configmap\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:18:31.433720 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.433548    2577 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-istio-kubeconfig\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:18:31.433720 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.433564    2577 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-cacerts\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:18:31.433720 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.433578    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h66p\" (UniqueName: \"kubernetes.io/projected/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-kube-api-access-2h66p\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:18:31.433720 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.433592    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-istio-csr-dns-cert\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:18:31.433720 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.433605    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490-istio-token\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:18:31.598527 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.598406    2577 generic.go:356] "Generic (PLEG): container finished" podID="b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490" containerID="3977268be5d759d46989aedac60bf107f8e022c4a0e0105366728e54c994d88a" exitCode=0
Sep 17 00:18:31.598886 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.598532    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"
Sep 17 00:18:31.598886 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.598564    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps" event={"ID":"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490","Type":"ContainerDied","Data":"3977268be5d759d46989aedac60bf107f8e022c4a0e0105366728e54c994d88a"}
Sep 17 00:18:31.598886 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.598638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps" event={"ID":"b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490","Type":"ContainerDied","Data":"2287e9dc9bd4c5374028fdb2f9ecb17bee774633f8f25f2aa30ff0bf4ea241b0"}
Sep 17 00:18:31.598886 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.598664    2577 scope.go:117] "RemoveContainer" containerID="3977268be5d759d46989aedac60bf107f8e022c4a0e0105366728e54c994d88a"
Sep 17 00:18:31.626979 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.626938    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"]
Sep 17 00:18:31.632490 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.632459    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lqnps"]
Sep 17 00:18:31.642084 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.642060    2577 scope.go:117] "RemoveContainer" containerID="3977268be5d759d46989aedac60bf107f8e022c4a0e0105366728e54c994d88a"
Sep 17 00:18:31.642501 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:18:31.642471    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3977268be5d759d46989aedac60bf107f8e022c4a0e0105366728e54c994d88a\": container with ID starting with 3977268be5d759d46989aedac60bf107f8e022c4a0e0105366728e54c994d88a not found: ID does not exist" containerID="3977268be5d759d46989aedac60bf107f8e022c4a0e0105366728e54c994d88a"
Sep 17 00:18:31.642727 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:31.642514    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3977268be5d759d46989aedac60bf107f8e022c4a0e0105366728e54c994d88a"} err="failed to get container status \"3977268be5d759d46989aedac60bf107f8e022c4a0e0105366728e54c994d88a\": rpc error: code = NotFound desc = could not find container \"3977268be5d759d46989aedac60bf107f8e022c4a0e0105366728e54c994d88a\": container with ID starting with 3977268be5d759d46989aedac60bf107f8e022c4a0e0105366728e54c994d88a not found: ID does not exist"
Sep 17 00:18:32.604662 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:32.604616    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mnjsh" event={"ID":"c85fdce2-ade0-4b20-b82d-0decac04bf20","Type":"ContainerStarted","Data":"8cde3962c5af223586b02a9b938695debf0ee4f200d12c734a3be626ea9b1707"}
Sep 17 00:18:32.605105 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:32.604824    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mnjsh"
Sep 17 00:18:32.620612 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:32.620564    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mnjsh" podStartSLOduration=1.73893192 podStartE2EDuration="4.620546726s" totalImagesPullingTime="2.881614806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:18:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:18:32.618983367 +0000 UTC m=+525.477476502" watchObservedRunningTime="2026-09-17 00:18:32.620546726 +0000 UTC m=+525.479039861"
Sep 17 00:18:33.607634 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:33.607599    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490" path="/var/lib/kubelet/pods/b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490/volumes"
Sep 17 00:18:34.615344 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:34.615309    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-96l8l" event={"ID":"e37901a4-81e2-41bb-bb2a-1acf51d61759","Type":"ContainerStarted","Data":"a6122f1ea3c987de631f45d010e9e7389fed72b1b5172772068f7af584126c93"}
Sep 17 00:18:34.615744 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:34.615451    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-96l8l"
Sep 17 00:18:34.631325 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:18:34.631269    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-96l8l" podStartSLOduration=3.383887424 podStartE2EDuration="6.631254233s" totalImagesPullingTime="3.247366809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:18:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:18:34.630213263 +0000 UTC m=+527.488706402" watchObservedRunningTime="2026-09-17 00:18:34.631254233 +0000 UTC m=+527.489747368"
Sep 17 00:19:00.722025 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:00.721994    2577 generic.go:356] "Generic (PLEG): container finished" podID="e37901a4-81e2-41bb-bb2a-1acf51d61759" containerID="a6122f1ea3c987de631f45d010e9e7389fed72b1b5172772068f7af584126c93" exitCode=1
Sep 17 00:19:00.722393 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:00.722069    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-96l8l" event={"ID":"e37901a4-81e2-41bb-bb2a-1acf51d61759","Type":"ContainerDied","Data":"a6122f1ea3c987de631f45d010e9e7389fed72b1b5172772068f7af584126c93"}
Sep 17 00:19:00.722455 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:00.722398    2577 scope.go:117] "RemoveContainer" containerID="a6122f1ea3c987de631f45d010e9e7389fed72b1b5172772068f7af584126c93"
Sep 17 00:19:01.728394 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:01.728362    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-96l8l" event={"ID":"e37901a4-81e2-41bb-bb2a-1acf51d61759","Type":"ContainerStarted","Data":"b5085278c2bd97ea47848c40f76193d70a3fc48fa10cd38c700bd8dac98fc0d6"}
Sep 17 00:19:01.728778 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:01.728578    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-96l8l"
Sep 17 00:19:03.615931 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:03.615895    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mnjsh"
Sep 17 00:19:32.734775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:32.734735    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-96l8l"
Sep 17 00:19:42.623527 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.623496    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v"]
Sep 17 00:19:42.623986 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.623779    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490" containerName="discovery"
Sep 17 00:19:42.623986 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.623790    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490" containerName="discovery"
Sep 17 00:19:42.623986 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.623842    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b33b9f0c-8f41-48e4-a6a3-a6e7b2b70490" containerName="discovery"
Sep 17 00:19:42.626757 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.626739    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v"
Sep 17 00:19:42.629018 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.628986    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-q8lnv\""
Sep 17 00:19:42.629151 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.629083    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 00:19:42.629995 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.629963    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 00:19:42.634183 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.634160    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v"]
Sep 17 00:19:42.724122 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.724090    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb"]
Sep 17 00:19:42.727329 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.727312    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb"
Sep 17 00:19:42.736949 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.736927    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb"]
Sep 17 00:19:42.763908 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.763881    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56752768-44da-47b0-9511-a9f580e24c3f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v\" (UID: \"56752768-44da-47b0-9511-a9f580e24c3f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v"
Sep 17 00:19:42.764068 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.763928    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvb42\" (UniqueName: \"kubernetes.io/projected/56752768-44da-47b0-9511-a9f580e24c3f-kube-api-access-fvb42\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v\" (UID: \"56752768-44da-47b0-9511-a9f580e24c3f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v"
Sep 17 00:19:42.764068 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.763980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56752768-44da-47b0-9511-a9f580e24c3f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v\" (UID: \"56752768-44da-47b0-9511-a9f580e24c3f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v"
Sep 17 00:19:42.819887 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.819855    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg"]
Sep 17 00:19:42.823177 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.823162    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg"
Sep 17 00:19:42.831115 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.831093    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg"]
Sep 17 00:19:42.864538 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.864500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvb42\" (UniqueName: \"kubernetes.io/projected/56752768-44da-47b0-9511-a9f580e24c3f-kube-api-access-fvb42\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v\" (UID: \"56752768-44da-47b0-9511-a9f580e24c3f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v"
Sep 17 00:19:42.864711 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.864549    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56752768-44da-47b0-9511-a9f580e24c3f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v\" (UID: \"56752768-44da-47b0-9511-a9f580e24c3f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v"
Sep 17 00:19:42.864711 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.864575    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7efc4d22-82f2-4691-86e5-0ba7556f06d3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb\" (UID: \"7efc4d22-82f2-4691-86e5-0ba7556f06d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb"
Sep 17 00:19:42.864711 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.864603    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7efc4d22-82f2-4691-86e5-0ba7556f06d3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb\" (UID: \"7efc4d22-82f2-4691-86e5-0ba7556f06d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb"
Sep 17 00:19:42.864711 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.864624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvfqs\" (UniqueName: \"kubernetes.io/projected/7efc4d22-82f2-4691-86e5-0ba7556f06d3-kube-api-access-nvfqs\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb\" (UID: \"7efc4d22-82f2-4691-86e5-0ba7556f06d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb"
Sep 17 00:19:42.864711 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.864646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56752768-44da-47b0-9511-a9f580e24c3f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v\" (UID: \"56752768-44da-47b0-9511-a9f580e24c3f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v"
Sep 17 00:19:42.865008 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.864967    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56752768-44da-47b0-9511-a9f580e24c3f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v\" (UID: \"56752768-44da-47b0-9511-a9f580e24c3f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v"
Sep 17 00:19:42.865008 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.864966    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56752768-44da-47b0-9511-a9f580e24c3f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v\" (UID: \"56752768-44da-47b0-9511-a9f580e24c3f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v"
Sep 17 00:19:42.872509 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.872485    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvb42\" (UniqueName: \"kubernetes.io/projected/56752768-44da-47b0-9511-a9f580e24c3f-kube-api-access-fvb42\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v\" (UID: \"56752768-44da-47b0-9511-a9f580e24c3f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v"
Sep 17 00:19:42.922366 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.922277    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7"]
Sep 17 00:19:42.925616 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.925600    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7"
Sep 17 00:19:42.933823 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.933797    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7"]
Sep 17 00:19:42.936452 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.936404    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v"
Sep 17 00:19:42.965310 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.965278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7efc4d22-82f2-4691-86e5-0ba7556f06d3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb\" (UID: \"7efc4d22-82f2-4691-86e5-0ba7556f06d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb"
Sep 17 00:19:42.965310 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.965315    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb077973-8e3f-4098-babe-069cd67127ce-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg\" (UID: \"eb077973-8e3f-4098-babe-069cd67127ce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg"
Sep 17 00:19:42.965562 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.965340    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvfqs\" (UniqueName: \"kubernetes.io/projected/7efc4d22-82f2-4691-86e5-0ba7556f06d3-kube-api-access-nvfqs\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb\" (UID: \"7efc4d22-82f2-4691-86e5-0ba7556f06d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb"
Sep 17 00:19:42.965562 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.965360    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb077973-8e3f-4098-babe-069cd67127ce-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg\" (UID: \"eb077973-8e3f-4098-babe-069cd67127ce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg"
Sep 17 00:19:42.965562 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.965405    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh66p\" (UniqueName: \"kubernetes.io/projected/eb077973-8e3f-4098-babe-069cd67127ce-kube-api-access-qh66p\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg\" (UID: \"eb077973-8e3f-4098-babe-069cd67127ce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg"
Sep 17 00:19:42.965562 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.965475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7efc4d22-82f2-4691-86e5-0ba7556f06d3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb\" (UID: \"7efc4d22-82f2-4691-86e5-0ba7556f06d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb"
Sep 17 00:19:42.965842 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.965770    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7efc4d22-82f2-4691-86e5-0ba7556f06d3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb\" (UID: \"7efc4d22-82f2-4691-86e5-0ba7556f06d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb"
Sep 17 00:19:42.965842 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.965769    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7efc4d22-82f2-4691-86e5-0ba7556f06d3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb\" (UID: \"7efc4d22-82f2-4691-86e5-0ba7556f06d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb"
Sep 17 00:19:42.975200 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:42.975174    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvfqs\" (UniqueName: \"kubernetes.io/projected/7efc4d22-82f2-4691-86e5-0ba7556f06d3-kube-api-access-nvfqs\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb\" (UID: \"7efc4d22-82f2-4691-86e5-0ba7556f06d3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb"
Sep 17 00:19:43.036740 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.036698    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb"
Sep 17 00:19:43.065983 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.065938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2zd5\" (UniqueName: \"kubernetes.io/projected/a6096034-0167-4d07-96de-7a4a86ef050d-kube-api-access-m2zd5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7\" (UID: \"a6096034-0167-4d07-96de-7a4a86ef050d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7"
Sep 17 00:19:43.065983 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.065993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh66p\" (UniqueName: \"kubernetes.io/projected/eb077973-8e3f-4098-babe-069cd67127ce-kube-api-access-qh66p\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg\" (UID: \"eb077973-8e3f-4098-babe-069cd67127ce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg"
Sep 17 00:19:43.066198 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.066038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb077973-8e3f-4098-babe-069cd67127ce-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg\" (UID: \"eb077973-8e3f-4098-babe-069cd67127ce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg"
Sep 17 00:19:43.066198 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.066081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6096034-0167-4d07-96de-7a4a86ef050d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7\" (UID: \"a6096034-0167-4d07-96de-7a4a86ef050d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7"
Sep 17 00:19:43.066198 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.066143    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6096034-0167-4d07-96de-7a4a86ef050d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7\" (UID: \"a6096034-0167-4d07-96de-7a4a86ef050d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7"
Sep 17 00:19:43.066198 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.066179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb077973-8e3f-4098-babe-069cd67127ce-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg\" (UID: \"eb077973-8e3f-4098-babe-069cd67127ce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg"
Sep 17 00:19:43.066398 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.066375    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb077973-8e3f-4098-babe-069cd67127ce-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg\" (UID: \"eb077973-8e3f-4098-babe-069cd67127ce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg"
Sep 17 00:19:43.066538 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.066517    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb077973-8e3f-4098-babe-069cd67127ce-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg\" (UID: \"eb077973-8e3f-4098-babe-069cd67127ce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg"
Sep 17 00:19:43.074391 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.074360    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh66p\" (UniqueName: \"kubernetes.io/projected/eb077973-8e3f-4098-babe-069cd67127ce-kube-api-access-qh66p\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg\" (UID: \"eb077973-8e3f-4098-babe-069cd67127ce\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg"
Sep 17 00:19:43.132402 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.132372    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg"
Sep 17 00:19:43.167556 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.167519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6096034-0167-4d07-96de-7a4a86ef050d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7\" (UID: \"a6096034-0167-4d07-96de-7a4a86ef050d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7"
Sep 17 00:19:43.167718 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.167561    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6096034-0167-4d07-96de-7a4a86ef050d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7\" (UID: \"a6096034-0167-4d07-96de-7a4a86ef050d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7"
Sep 17 00:19:43.167718 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.167702    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2zd5\" (UniqueName: \"kubernetes.io/projected/a6096034-0167-4d07-96de-7a4a86ef050d-kube-api-access-m2zd5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7\" (UID: \"a6096034-0167-4d07-96de-7a4a86ef050d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7"
Sep 17 00:19:43.167873 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.167864    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6096034-0167-4d07-96de-7a4a86ef050d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7\" (UID: \"a6096034-0167-4d07-96de-7a4a86ef050d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7"
Sep 17 00:19:43.167964 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.167939    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6096034-0167-4d07-96de-7a4a86ef050d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7\" (UID: \"a6096034-0167-4d07-96de-7a4a86ef050d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7"
Sep 17 00:19:43.177373 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.177292    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2zd5\" (UniqueName: \"kubernetes.io/projected/a6096034-0167-4d07-96de-7a4a86ef050d-kube-api-access-m2zd5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7\" (UID: \"a6096034-0167-4d07-96de-7a4a86ef050d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7"
Sep 17 00:19:43.209120 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.209092    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v"]
Sep 17 00:19:43.210715 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:19:43.210692    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56752768_44da_47b0_9511_a9f580e24c3f.slice/crio-7631f67837c5bd2d8f132ad68bf4c480f63c0e37139c2404b305d1be2346014f WatchSource:0}: Error finding container 7631f67837c5bd2d8f132ad68bf4c480f63c0e37139c2404b305d1be2346014f: Status 404 returned error can't find the container with id 7631f67837c5bd2d8f132ad68bf4c480f63c0e37139c2404b305d1be2346014f
Sep 17 00:19:43.235343 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.235323    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7"
Sep 17 00:19:43.311144 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.311113    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb"]
Sep 17 00:19:43.341198 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:19:43.341169    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7efc4d22_82f2_4691_86e5_0ba7556f06d3.slice/crio-f91dec8706edd4ed12a7b6bd33f7c2d734fe44bcecb59c0d94e730cfb315f0df WatchSource:0}: Error finding container f91dec8706edd4ed12a7b6bd33f7c2d734fe44bcecb59c0d94e730cfb315f0df: Status 404 returned error can't find the container with id f91dec8706edd4ed12a7b6bd33f7c2d734fe44bcecb59c0d94e730cfb315f0df
Sep 17 00:19:43.404300 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.404226    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg"]
Sep 17 00:19:43.407711 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:19:43.407689    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb077973_8e3f_4098_babe_069cd67127ce.slice/crio-ceb9d2f1aa95253034b0d6a29103dd9681d3dbf7b7a454c00ad0fcab813f67e8 WatchSource:0}: Error finding container ceb9d2f1aa95253034b0d6a29103dd9681d3dbf7b7a454c00ad0fcab813f67e8: Status 404 returned error can't find the container with id ceb9d2f1aa95253034b0d6a29103dd9681d3dbf7b7a454c00ad0fcab813f67e8
Sep 17 00:19:43.504263 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.504241    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7"]
Sep 17 00:19:43.507147 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:19:43.507117    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6096034_0167_4d07_96de_7a4a86ef050d.slice/crio-bd39ce73d97099b99c87571b3a668e191479a9854e875e480a6e90ab2ef42ef4 WatchSource:0}: Error finding container bd39ce73d97099b99c87571b3a668e191479a9854e875e480a6e90ab2ef42ef4: Status 404 returned error can't find the container with id bd39ce73d97099b99c87571b3a668e191479a9854e875e480a6e90ab2ef42ef4
Sep 17 00:19:43.913614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.913529    2577 generic.go:356] "Generic (PLEG): container finished" podID="a6096034-0167-4d07-96de-7a4a86ef050d" containerID="b40c4670820fd917b79c31db773ebfcb260850768015c3b74e212ecc93007294" exitCode=0
Sep 17 00:19:43.914023 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.913611    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7" event={"ID":"a6096034-0167-4d07-96de-7a4a86ef050d","Type":"ContainerDied","Data":"b40c4670820fd917b79c31db773ebfcb260850768015c3b74e212ecc93007294"}
Sep 17 00:19:43.914023 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.913665    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7" event={"ID":"a6096034-0167-4d07-96de-7a4a86ef050d","Type":"ContainerStarted","Data":"bd39ce73d97099b99c87571b3a668e191479a9854e875e480a6e90ab2ef42ef4"}
Sep 17 00:19:43.915182 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.915159    2577 generic.go:356] "Generic (PLEG): container finished" podID="56752768-44da-47b0-9511-a9f580e24c3f" containerID="cd45e8518dfbb6bce8e2cb084c67091c9d346f0b74df13fe224e105d44a10ee1" exitCode=0
Sep 17 00:19:43.915280 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.915252    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v" event={"ID":"56752768-44da-47b0-9511-a9f580e24c3f","Type":"ContainerDied","Data":"cd45e8518dfbb6bce8e2cb084c67091c9d346f0b74df13fe224e105d44a10ee1"}
Sep 17 00:19:43.915333 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.915295    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v" event={"ID":"56752768-44da-47b0-9511-a9f580e24c3f","Type":"ContainerStarted","Data":"7631f67837c5bd2d8f132ad68bf4c480f63c0e37139c2404b305d1be2346014f"}
Sep 17 00:19:43.917053 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.917032    2577 generic.go:356] "Generic (PLEG): container finished" podID="eb077973-8e3f-4098-babe-069cd67127ce" containerID="34bd26fc6e99e59ebe7af47b50f00db66f6de062f77c3cc075b3f7365f7aab2f" exitCode=0
Sep 17 00:19:43.917151 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.917112    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg" event={"ID":"eb077973-8e3f-4098-babe-069cd67127ce","Type":"ContainerDied","Data":"34bd26fc6e99e59ebe7af47b50f00db66f6de062f77c3cc075b3f7365f7aab2f"}
Sep 17 00:19:43.917204 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.917148    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg" event={"ID":"eb077973-8e3f-4098-babe-069cd67127ce","Type":"ContainerStarted","Data":"ceb9d2f1aa95253034b0d6a29103dd9681d3dbf7b7a454c00ad0fcab813f67e8"}
Sep 17 00:19:43.919002 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.918980    2577 generic.go:356] "Generic (PLEG): container finished" podID="7efc4d22-82f2-4691-86e5-0ba7556f06d3" containerID="7a29eab55c59fffbf278751ed79eb33c9808be48ba9cfbc496370f5428cff3b5" exitCode=0
Sep 17 00:19:43.919146 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.919053    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb" event={"ID":"7efc4d22-82f2-4691-86e5-0ba7556f06d3","Type":"ContainerDied","Data":"7a29eab55c59fffbf278751ed79eb33c9808be48ba9cfbc496370f5428cff3b5"}
Sep 17 00:19:43.919146 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:43.919092    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb" event={"ID":"7efc4d22-82f2-4691-86e5-0ba7556f06d3","Type":"ContainerStarted","Data":"f91dec8706edd4ed12a7b6bd33f7c2d734fe44bcecb59c0d94e730cfb315f0df"}
Sep 17 00:19:44.925441 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:44.925396    2577 generic.go:356] "Generic (PLEG): container finished" podID="56752768-44da-47b0-9511-a9f580e24c3f" containerID="a579ae06d8afd3bd3e5d10dc5b8889178c7431be0e9aa91dd5d99a3f25777d70" exitCode=0
Sep 17 00:19:44.925800 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:44.925447    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v" event={"ID":"56752768-44da-47b0-9511-a9f580e24c3f","Type":"ContainerDied","Data":"a579ae06d8afd3bd3e5d10dc5b8889178c7431be0e9aa91dd5d99a3f25777d70"}
Sep 17 00:19:45.931977 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:45.931942    2577 generic.go:356] "Generic (PLEG): container finished" podID="a6096034-0167-4d07-96de-7a4a86ef050d" containerID="5aaea35e801ef9b58e9115be28c70bad627a57e681b6b36fd7534eed6e294697" exitCode=0
Sep 17 00:19:45.932407 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:45.932015    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7" event={"ID":"a6096034-0167-4d07-96de-7a4a86ef050d","Type":"ContainerDied","Data":"5aaea35e801ef9b58e9115be28c70bad627a57e681b6b36fd7534eed6e294697"}
Sep 17 00:19:45.934287 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:45.934265    2577 generic.go:356] "Generic (PLEG): container finished" podID="56752768-44da-47b0-9511-a9f580e24c3f" containerID="beade8676877042549c0965ab51059cd0773d736c85861afb2232df9c81b7509" exitCode=0
Sep 17 00:19:45.934408 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:45.934386    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v" event={"ID":"56752768-44da-47b0-9511-a9f580e24c3f","Type":"ContainerDied","Data":"beade8676877042549c0965ab51059cd0773d736c85861afb2232df9c81b7509"}
Sep 17 00:19:45.936196 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:45.936177    2577 generic.go:356] "Generic (PLEG): container finished" podID="eb077973-8e3f-4098-babe-069cd67127ce" containerID="76c89db8e1217cc7fc5dcf28c6a12de01487989b6aedd419066701f71e604652" exitCode=0
Sep 17 00:19:45.936293 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:45.936240    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg" event={"ID":"eb077973-8e3f-4098-babe-069cd67127ce","Type":"ContainerDied","Data":"76c89db8e1217cc7fc5dcf28c6a12de01487989b6aedd419066701f71e604652"}
Sep 17 00:19:45.938239 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:45.938217    2577 generic.go:356] "Generic (PLEG): container finished" podID="7efc4d22-82f2-4691-86e5-0ba7556f06d3" containerID="63350ca566d6d602cff88b99e1e82c1ec68e106cd4c073f9abb3771027705911" exitCode=0
Sep 17 00:19:45.938329 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:45.938249    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb" event={"ID":"7efc4d22-82f2-4691-86e5-0ba7556f06d3","Type":"ContainerDied","Data":"63350ca566d6d602cff88b99e1e82c1ec68e106cd4c073f9abb3771027705911"}
Sep 17 00:19:46.944931 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:46.944895    2577 generic.go:356] "Generic (PLEG): container finished" podID="a6096034-0167-4d07-96de-7a4a86ef050d" containerID="2819e3c8ecb320216d7ad2f2c8357e056bb1c36d7573fb9426d1abc9a9593eb4" exitCode=0
Sep 17 00:19:46.945280 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:46.944974    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7" event={"ID":"a6096034-0167-4d07-96de-7a4a86ef050d","Type":"ContainerDied","Data":"2819e3c8ecb320216d7ad2f2c8357e056bb1c36d7573fb9426d1abc9a9593eb4"}
Sep 17 00:19:46.946841 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:46.946820    2577 generic.go:356] "Generic (PLEG): container finished" podID="eb077973-8e3f-4098-babe-069cd67127ce" containerID="f853d0f42743788de1fcf6ec68ff1007cc0b40d9c487b3a37e66f99d258bdd51" exitCode=0
Sep 17 00:19:46.946955 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:46.946880    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg" event={"ID":"eb077973-8e3f-4098-babe-069cd67127ce","Type":"ContainerDied","Data":"f853d0f42743788de1fcf6ec68ff1007cc0b40d9c487b3a37e66f99d258bdd51"}
Sep 17 00:19:46.948855 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:46.948835    2577 generic.go:356] "Generic (PLEG): container finished" podID="7efc4d22-82f2-4691-86e5-0ba7556f06d3" containerID="bc7185709ca08eec88c56fdf3d410c96d25fa0f6bbf6c18f1df4ce018ef8da41" exitCode=0
Sep 17 00:19:46.948940 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:46.948921    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb" event={"ID":"7efc4d22-82f2-4691-86e5-0ba7556f06d3","Type":"ContainerDied","Data":"bc7185709ca08eec88c56fdf3d410c96d25fa0f6bbf6c18f1df4ce018ef8da41"}
Sep 17 00:19:47.072739 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:47.072714    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v"
Sep 17 00:19:47.197544 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:47.197464    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvb42\" (UniqueName: \"kubernetes.io/projected/56752768-44da-47b0-9511-a9f580e24c3f-kube-api-access-fvb42\") pod \"56752768-44da-47b0-9511-a9f580e24c3f\" (UID: \"56752768-44da-47b0-9511-a9f580e24c3f\") "
Sep 17 00:19:47.197544 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:47.197520    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56752768-44da-47b0-9511-a9f580e24c3f-bundle\") pod \"56752768-44da-47b0-9511-a9f580e24c3f\" (UID: \"56752768-44da-47b0-9511-a9f580e24c3f\") "
Sep 17 00:19:47.197768 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:47.197586    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56752768-44da-47b0-9511-a9f580e24c3f-util\") pod \"56752768-44da-47b0-9511-a9f580e24c3f\" (UID: \"56752768-44da-47b0-9511-a9f580e24c3f\") "
Sep 17 00:19:47.198075 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:47.198038    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56752768-44da-47b0-9511-a9f580e24c3f-bundle" (OuterVolumeSpecName: "bundle") pod "56752768-44da-47b0-9511-a9f580e24c3f" (UID: "56752768-44da-47b0-9511-a9f580e24c3f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:19:47.199746 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:47.199725    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56752768-44da-47b0-9511-a9f580e24c3f-kube-api-access-fvb42" (OuterVolumeSpecName: "kube-api-access-fvb42") pod "56752768-44da-47b0-9511-a9f580e24c3f" (UID: "56752768-44da-47b0-9511-a9f580e24c3f"). InnerVolumeSpecName "kube-api-access-fvb42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:19:47.203228 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:47.203205    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56752768-44da-47b0-9511-a9f580e24c3f-util" (OuterVolumeSpecName: "util") pod "56752768-44da-47b0-9511-a9f580e24c3f" (UID: "56752768-44da-47b0-9511-a9f580e24c3f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:19:47.298305 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:47.298271    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56752768-44da-47b0-9511-a9f580e24c3f-bundle\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:19:47.298305 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:47.298299    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56752768-44da-47b0-9511-a9f580e24c3f-util\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:19:47.298305 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:47.298309    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvb42\" (UniqueName: \"kubernetes.io/projected/56752768-44da-47b0-9511-a9f580e24c3f-kube-api-access-fvb42\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:19:47.552788 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:47.552717    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/ovn-acl-logging/0.log"
Sep 17 00:19:47.552938 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:47.552719    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/ovn-acl-logging/0.log"
Sep 17 00:19:47.960991 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:47.960955    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v" event={"ID":"56752768-44da-47b0-9511-a9f580e24c3f","Type":"ContainerDied","Data":"7631f67837c5bd2d8f132ad68bf4c480f63c0e37139c2404b305d1be2346014f"}
Sep 17 00:19:47.961406 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:47.960992    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kpc8v"
Sep 17 00:19:47.961406 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:47.961037    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7631f67837c5bd2d8f132ad68bf4c480f63c0e37139c2404b305d1be2346014f"
Sep 17 00:19:48.098003 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.097973    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7"
Sep 17 00:19:48.146720 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.146699    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb"
Sep 17 00:19:48.149176 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.149155    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg"
Sep 17 00:19:48.205591 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.205562    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2zd5\" (UniqueName: \"kubernetes.io/projected/a6096034-0167-4d07-96de-7a4a86ef050d-kube-api-access-m2zd5\") pod \"a6096034-0167-4d07-96de-7a4a86ef050d\" (UID: \"a6096034-0167-4d07-96de-7a4a86ef050d\") "
Sep 17 00:19:48.205723 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.205652    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6096034-0167-4d07-96de-7a4a86ef050d-util\") pod \"a6096034-0167-4d07-96de-7a4a86ef050d\" (UID: \"a6096034-0167-4d07-96de-7a4a86ef050d\") "
Sep 17 00:19:48.205723 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.205682    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6096034-0167-4d07-96de-7a4a86ef050d-bundle\") pod \"a6096034-0167-4d07-96de-7a4a86ef050d\" (UID: \"a6096034-0167-4d07-96de-7a4a86ef050d\") "
Sep 17 00:19:48.206206 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.206179    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6096034-0167-4d07-96de-7a4a86ef050d-bundle" (OuterVolumeSpecName: "bundle") pod "a6096034-0167-4d07-96de-7a4a86ef050d" (UID: "a6096034-0167-4d07-96de-7a4a86ef050d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:19:48.207600 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.207577    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6096034-0167-4d07-96de-7a4a86ef050d-kube-api-access-m2zd5" (OuterVolumeSpecName: "kube-api-access-m2zd5") pod "a6096034-0167-4d07-96de-7a4a86ef050d" (UID: "a6096034-0167-4d07-96de-7a4a86ef050d"). InnerVolumeSpecName "kube-api-access-m2zd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:19:48.210849 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.210824    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6096034-0167-4d07-96de-7a4a86ef050d-util" (OuterVolumeSpecName: "util") pod "a6096034-0167-4d07-96de-7a4a86ef050d" (UID: "a6096034-0167-4d07-96de-7a4a86ef050d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:19:48.307044 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.306956    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7efc4d22-82f2-4691-86e5-0ba7556f06d3-util\") pod \"7efc4d22-82f2-4691-86e5-0ba7556f06d3\" (UID: \"7efc4d22-82f2-4691-86e5-0ba7556f06d3\") "
Sep 17 00:19:48.307044 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.307016    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvfqs\" (UniqueName: \"kubernetes.io/projected/7efc4d22-82f2-4691-86e5-0ba7556f06d3-kube-api-access-nvfqs\") pod \"7efc4d22-82f2-4691-86e5-0ba7556f06d3\" (UID: \"7efc4d22-82f2-4691-86e5-0ba7556f06d3\") "
Sep 17 00:19:48.307268 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.307073    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb077973-8e3f-4098-babe-069cd67127ce-bundle\") pod \"eb077973-8e3f-4098-babe-069cd67127ce\" (UID: \"eb077973-8e3f-4098-babe-069cd67127ce\") "
Sep 17 00:19:48.307268 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.307108    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb077973-8e3f-4098-babe-069cd67127ce-util\") pod \"eb077973-8e3f-4098-babe-069cd67127ce\" (UID: \"eb077973-8e3f-4098-babe-069cd67127ce\") "
Sep 17 00:19:48.307373 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.307268    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7efc4d22-82f2-4691-86e5-0ba7556f06d3-bundle\") pod \"7efc4d22-82f2-4691-86e5-0ba7556f06d3\" (UID: \"7efc4d22-82f2-4691-86e5-0ba7556f06d3\") "
Sep 17 00:19:48.307373 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.307321    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh66p\" (UniqueName: \"kubernetes.io/projected/eb077973-8e3f-4098-babe-069cd67127ce-kube-api-access-qh66p\") pod \"eb077973-8e3f-4098-babe-069cd67127ce\" (UID: \"eb077973-8e3f-4098-babe-069cd67127ce\") "
Sep 17 00:19:48.307565 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.307543    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6096034-0167-4d07-96de-7a4a86ef050d-util\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:19:48.307637 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.307568    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6096034-0167-4d07-96de-7a4a86ef050d-bundle\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:19:48.307637 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.307582    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2zd5\" (UniqueName: \"kubernetes.io/projected/a6096034-0167-4d07-96de-7a4a86ef050d-kube-api-access-m2zd5\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:19:48.307743 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.307713    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7efc4d22-82f2-4691-86e5-0ba7556f06d3-bundle" (OuterVolumeSpecName: "bundle") pod "7efc4d22-82f2-4691-86e5-0ba7556f06d3" (UID: "7efc4d22-82f2-4691-86e5-0ba7556f06d3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:19:48.307905 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.307883    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb077973-8e3f-4098-babe-069cd67127ce-bundle" (OuterVolumeSpecName: "bundle") pod "eb077973-8e3f-4098-babe-069cd67127ce" (UID: "eb077973-8e3f-4098-babe-069cd67127ce"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:19:48.309264 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.309242    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7efc4d22-82f2-4691-86e5-0ba7556f06d3-kube-api-access-nvfqs" (OuterVolumeSpecName: "kube-api-access-nvfqs") pod "7efc4d22-82f2-4691-86e5-0ba7556f06d3" (UID: "7efc4d22-82f2-4691-86e5-0ba7556f06d3"). InnerVolumeSpecName "kube-api-access-nvfqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:19:48.309359 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.309308    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb077973-8e3f-4098-babe-069cd67127ce-kube-api-access-qh66p" (OuterVolumeSpecName: "kube-api-access-qh66p") pod "eb077973-8e3f-4098-babe-069cd67127ce" (UID: "eb077973-8e3f-4098-babe-069cd67127ce"). InnerVolumeSpecName "kube-api-access-qh66p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:19:48.316084 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.316061    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb077973-8e3f-4098-babe-069cd67127ce-util" (OuterVolumeSpecName: "util") pod "eb077973-8e3f-4098-babe-069cd67127ce" (UID: "eb077973-8e3f-4098-babe-069cd67127ce"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:19:48.316169 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.316152    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7efc4d22-82f2-4691-86e5-0ba7556f06d3-util" (OuterVolumeSpecName: "util") pod "7efc4d22-82f2-4691-86e5-0ba7556f06d3" (UID: "7efc4d22-82f2-4691-86e5-0ba7556f06d3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:19:48.408770 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.408735    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh66p\" (UniqueName: \"kubernetes.io/projected/eb077973-8e3f-4098-babe-069cd67127ce-kube-api-access-qh66p\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:19:48.408770 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.408764    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7efc4d22-82f2-4691-86e5-0ba7556f06d3-util\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:19:48.408770 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.408774    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvfqs\" (UniqueName: \"kubernetes.io/projected/7efc4d22-82f2-4691-86e5-0ba7556f06d3-kube-api-access-nvfqs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:19:48.409005 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.408783    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb077973-8e3f-4098-babe-069cd67127ce-bundle\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:19:48.409005 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.408791    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb077973-8e3f-4098-babe-069cd67127ce-util\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:19:48.409005 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.408798    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7efc4d22-82f2-4691-86e5-0ba7556f06d3-bundle\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:19:48.967654 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.967621    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg" event={"ID":"eb077973-8e3f-4098-babe-069cd67127ce","Type":"ContainerDied","Data":"ceb9d2f1aa95253034b0d6a29103dd9681d3dbf7b7a454c00ad0fcab813f67e8"}
Sep 17 00:19:48.968087 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.967671    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ceb9d2f1aa95253034b0d6a29103dd9681d3dbf7b7a454c00ad0fcab813f67e8"
Sep 17 00:19:48.968087 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.967631    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avc4zg"
Sep 17 00:19:48.969704 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.969681    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb"
Sep 17 00:19:48.969808 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.969681    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67gggb" event={"ID":"7efc4d22-82f2-4691-86e5-0ba7556f06d3","Type":"ContainerDied","Data":"f91dec8706edd4ed12a7b6bd33f7c2d734fe44bcecb59c0d94e730cfb315f0df"}
Sep 17 00:19:48.969808 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.969804    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f91dec8706edd4ed12a7b6bd33f7c2d734fe44bcecb59c0d94e730cfb315f0df"
Sep 17 00:19:48.971547 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.971529    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7" event={"ID":"a6096034-0167-4d07-96de-7a4a86ef050d","Type":"ContainerDied","Data":"bd39ce73d97099b99c87571b3a668e191479a9854e875e480a6e90ab2ef42ef4"}
Sep 17 00:19:48.971657 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.971557    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd39ce73d97099b99c87571b3a668e191479a9854e875e480a6e90ab2ef42ef4"
Sep 17 00:19:48.971657 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:48.971557    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckwcj7"
Sep 17 00:19:55.431191 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431147    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-kqwmq"]
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431463    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56752768-44da-47b0-9511-a9f580e24c3f" containerName="pull"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431480    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="56752768-44da-47b0-9511-a9f580e24c3f" containerName="pull"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431492    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6096034-0167-4d07-96de-7a4a86ef050d" containerName="extract"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431499    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6096034-0167-4d07-96de-7a4a86ef050d" containerName="extract"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431509    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7efc4d22-82f2-4691-86e5-0ba7556f06d3" containerName="pull"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431515    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7efc4d22-82f2-4691-86e5-0ba7556f06d3" containerName="pull"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431524    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb077973-8e3f-4098-babe-069cd67127ce" containerName="util"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431529    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb077973-8e3f-4098-babe-069cd67127ce" containerName="util"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431535    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb077973-8e3f-4098-babe-069cd67127ce" containerName="extract"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431540    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb077973-8e3f-4098-babe-069cd67127ce" containerName="extract"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431549    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6096034-0167-4d07-96de-7a4a86ef050d" containerName="util"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431554    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6096034-0167-4d07-96de-7a4a86ef050d" containerName="util"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431565    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7efc4d22-82f2-4691-86e5-0ba7556f06d3" containerName="extract"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431574    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7efc4d22-82f2-4691-86e5-0ba7556f06d3" containerName="extract"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431584    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6096034-0167-4d07-96de-7a4a86ef050d" containerName="pull"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431589    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6096034-0167-4d07-96de-7a4a86ef050d" containerName="pull"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431597    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56752768-44da-47b0-9511-a9f580e24c3f" containerName="extract"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431603    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="56752768-44da-47b0-9511-a9f580e24c3f" containerName="extract"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431609    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56752768-44da-47b0-9511-a9f580e24c3f" containerName="util"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431614    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="56752768-44da-47b0-9511-a9f580e24c3f" containerName="util"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431620    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7efc4d22-82f2-4691-86e5-0ba7556f06d3" containerName="util"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431625    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7efc4d22-82f2-4691-86e5-0ba7556f06d3" containerName="util"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431631    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb077973-8e3f-4098-babe-069cd67127ce" containerName="pull"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431636    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb077973-8e3f-4098-babe-069cd67127ce" containerName="pull"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431693    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="56752768-44da-47b0-9511-a9f580e24c3f" containerName="extract"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431701    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6096034-0167-4d07-96de-7a4a86ef050d" containerName="extract"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431705    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7efc4d22-82f2-4691-86e5-0ba7556f06d3" containerName="extract"
Sep 17 00:19:55.433614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.431711    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb077973-8e3f-4098-babe-069cd67127ce" containerName="extract"
Sep 17 00:19:55.434534 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.434517    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-kqwmq"
Sep 17 00:19:55.437153 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.437127    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-ssvpz\""
Sep 17 00:19:55.437278 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.437129    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 00:19:55.438279 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.438261    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 00:19:55.444374 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.444350    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-kqwmq"]
Sep 17 00:19:55.453660 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.453629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wphw\" (UniqueName: \"kubernetes.io/projected/bc5b38a8-e333-4379-ad37-ddbfa0744717-kube-api-access-6wphw\") pod \"authorino-operator-76b478bd78-kqwmq\" (UID: \"bc5b38a8-e333-4379-ad37-ddbfa0744717\") " pod="kuadrant-system/authorino-operator-76b478bd78-kqwmq"
Sep 17 00:19:55.554702 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.554665    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wphw\" (UniqueName: \"kubernetes.io/projected/bc5b38a8-e333-4379-ad37-ddbfa0744717-kube-api-access-6wphw\") pod \"authorino-operator-76b478bd78-kqwmq\" (UID: \"bc5b38a8-e333-4379-ad37-ddbfa0744717\") " pod="kuadrant-system/authorino-operator-76b478bd78-kqwmq"
Sep 17 00:19:55.568348 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.568308    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wphw\" (UniqueName: \"kubernetes.io/projected/bc5b38a8-e333-4379-ad37-ddbfa0744717-kube-api-access-6wphw\") pod \"authorino-operator-76b478bd78-kqwmq\" (UID: \"bc5b38a8-e333-4379-ad37-ddbfa0744717\") " pod="kuadrant-system/authorino-operator-76b478bd78-kqwmq"
Sep 17 00:19:55.745593 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:55.745510    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-kqwmq"
Sep 17 00:19:56.019537 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:56.019465    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-kqwmq"]
Sep 17 00:19:56.021720 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:19:56.021689    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc5b38a8_e333_4379_ad37_ddbfa0744717.slice/crio-af51a77b555be319fea714d21ccab26fc5ee7fbd5cdcd7d146b22ceee3535b4f WatchSource:0}: Error finding container af51a77b555be319fea714d21ccab26fc5ee7fbd5cdcd7d146b22ceee3535b4f: Status 404 returned error can't find the container with id af51a77b555be319fea714d21ccab26fc5ee7fbd5cdcd7d146b22ceee3535b4f
Sep 17 00:19:57.009482 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:57.009410    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-kqwmq" event={"ID":"bc5b38a8-e333-4379-ad37-ddbfa0744717","Type":"ContainerStarted","Data":"af51a77b555be319fea714d21ccab26fc5ee7fbd5cdcd7d146b22ceee3535b4f"}
Sep 17 00:19:57.631206 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:57.631167    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mxsms"]
Sep 17 00:19:57.634756 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:57.634730    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mxsms"
Sep 17 00:19:57.637141 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:57.637116    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tlqf8\""
Sep 17 00:19:57.645280 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:57.645255    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mxsms"]
Sep 17 00:19:57.669054 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:57.669017    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrw62\" (UniqueName: \"kubernetes.io/projected/6cf42944-26fd-4299-99bc-3fd091be765f-kube-api-access-vrw62\") pod \"limitador-operator-controller-manager-74495c69f8-mxsms\" (UID: \"6cf42944-26fd-4299-99bc-3fd091be765f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mxsms"
Sep 17 00:19:57.770107 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:57.770063    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrw62\" (UniqueName: \"kubernetes.io/projected/6cf42944-26fd-4299-99bc-3fd091be765f-kube-api-access-vrw62\") pod \"limitador-operator-controller-manager-74495c69f8-mxsms\" (UID: \"6cf42944-26fd-4299-99bc-3fd091be765f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mxsms"
Sep 17 00:19:57.781941 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:57.781905    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrw62\" (UniqueName: \"kubernetes.io/projected/6cf42944-26fd-4299-99bc-3fd091be765f-kube-api-access-vrw62\") pod \"limitador-operator-controller-manager-74495c69f8-mxsms\" (UID: \"6cf42944-26fd-4299-99bc-3fd091be765f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mxsms"
Sep 17 00:19:57.948491 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:57.948454    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mxsms"
Sep 17 00:19:58.229684 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:19:58.229650    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mxsms"]
Sep 17 00:19:59.163247 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:19:59.163206    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cf42944_26fd_4299_99bc_3fd091be765f.slice/crio-c19a98e3d251cc75018486b5a62fbc39d4baac672f675ddbed13f9221148f7d9 WatchSource:0}: Error finding container c19a98e3d251cc75018486b5a62fbc39d4baac672f675ddbed13f9221148f7d9: Status 404 returned error can't find the container with id c19a98e3d251cc75018486b5a62fbc39d4baac672f675ddbed13f9221148f7d9
Sep 17 00:20:00.030690 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:00.030650    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mxsms" event={"ID":"6cf42944-26fd-4299-99bc-3fd091be765f","Type":"ContainerStarted","Data":"c19a98e3d251cc75018486b5a62fbc39d4baac672f675ddbed13f9221148f7d9"}
Sep 17 00:20:00.032297 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:00.032271    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-kqwmq" event={"ID":"bc5b38a8-e333-4379-ad37-ddbfa0744717","Type":"ContainerStarted","Data":"a64ba047a464a482b50168a8239f018a50d4a0f3870a4d7a1025e37decc98803"}
Sep 17 00:20:00.032440 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:00.032404    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-kqwmq"
Sep 17 00:20:00.056478 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:00.056407    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-kqwmq" podStartSLOduration=1.8538764909999998 podStartE2EDuration="5.056390968s" totalImagesPullingTime="3.202514477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:20:00.055487704 +0000 UTC m=+612.913980840" watchObservedRunningTime="2026-09-17 00:20:00.056390968 +0000 UTC m=+612.914884143"
Sep 17 00:20:01.879841 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:01.879758    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn"]
Sep 17 00:20:01.882962 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:01.882937    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn"
Sep 17 00:20:01.885361 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:01.885337    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-n26np\""
Sep 17 00:20:01.896786 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:01.896756    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn"]
Sep 17 00:20:01.897615 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:01.897587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2qns\" (UniqueName: \"kubernetes.io/projected/81252db6-9464-412c-8fd6-6bb866d67949-kube-api-access-f2qns\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9l4dn\" (UID: \"81252db6-9464-412c-8fd6-6bb866d67949\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn"
Sep 17 00:20:01.897745 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:01.897688    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81252db6-9464-412c-8fd6-6bb866d67949-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9l4dn\" (UID: \"81252db6-9464-412c-8fd6-6bb866d67949\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn"
Sep 17 00:20:01.998221 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:01.998169    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81252db6-9464-412c-8fd6-6bb866d67949-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9l4dn\" (UID: \"81252db6-9464-412c-8fd6-6bb866d67949\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn"
Sep 17 00:20:01.998406 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:01.998258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2qns\" (UniqueName: \"kubernetes.io/projected/81252db6-9464-412c-8fd6-6bb866d67949-kube-api-access-f2qns\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9l4dn\" (UID: \"81252db6-9464-412c-8fd6-6bb866d67949\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn"
Sep 17 00:20:01.998623 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:01.998598    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81252db6-9464-412c-8fd6-6bb866d67949-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9l4dn\" (UID: \"81252db6-9464-412c-8fd6-6bb866d67949\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn"
Sep 17 00:20:02.009263 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:02.009225    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2qns\" (UniqueName: \"kubernetes.io/projected/81252db6-9464-412c-8fd6-6bb866d67949-kube-api-access-f2qns\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9l4dn\" (UID: \"81252db6-9464-412c-8fd6-6bb866d67949\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn"
Sep 17 00:20:02.043012 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:02.042968    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mxsms" event={"ID":"6cf42944-26fd-4299-99bc-3fd091be765f","Type":"ContainerStarted","Data":"56f4b155a45d22383d9b7d7774accaa84282d5e91c47d799038f99cb580c7731"}
Sep 17 00:20:02.043200 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:02.043115    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mxsms"
Sep 17 00:20:02.061833 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:02.061749    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mxsms" podStartSLOduration=2.627996982 podStartE2EDuration="5.061730368s" totalImagesPullingTime="2.433733386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:19:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:20:02.061653285 +0000 UTC m=+614.920146430" watchObservedRunningTime="2026-09-17 00:20:02.061730368 +0000 UTC m=+614.920223496"
Sep 17 00:20:02.192982 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:02.192948    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn"
Sep 17 00:20:02.479163 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:02.479135    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn"]
Sep 17 00:20:02.481046 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:20:02.481019    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81252db6_9464_412c_8fd6_6bb866d67949.slice/crio-3f26b09351e1c1b0a81253fff827f4b247b088d9d86e632ef5a1f5ea5bf9df74 WatchSource:0}: Error finding container 3f26b09351e1c1b0a81253fff827f4b247b088d9d86e632ef5a1f5ea5bf9df74: Status 404 returned error can't find the container with id 3f26b09351e1c1b0a81253fff827f4b247b088d9d86e632ef5a1f5ea5bf9df74
Sep 17 00:20:03.055035 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:03.054994    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn" event={"ID":"81252db6-9464-412c-8fd6-6bb866d67949","Type":"ContainerStarted","Data":"3f26b09351e1c1b0a81253fff827f4b247b088d9d86e632ef5a1f5ea5bf9df74"}
Sep 17 00:20:08.084242 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:08.084210    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn" event={"ID":"81252db6-9464-412c-8fd6-6bb866d67949","Type":"ContainerStarted","Data":"c2337c4c475de96e3fd36ec793c5f9aa03a020c2fd0930f13d27d05cb3b65cf9"}
Sep 17 00:20:08.084621 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:08.084310    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn"
Sep 17 00:20:08.102564 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:08.102512    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn" podStartSLOduration=1.590847479 podStartE2EDuration="7.102496165s" totalImagesPullingTime="5.511648686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:20:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:20:08.101086585 +0000 UTC m=+620.959579719" watchObservedRunningTime="2026-09-17 00:20:08.102496165 +0000 UTC m=+620.960989301"
Sep 17 00:20:10.586221 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:10.586133    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-f5gwq"]
Sep 17 00:20:10.589456 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:10.589432    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-f5gwq"
Sep 17 00:20:10.591882 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:10.591854    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 17 00:20:10.592110 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:10.592088    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 17 00:20:10.592205 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:10.592187    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kqb42\""
Sep 17 00:20:10.596601 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:10.596579    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-f5gwq"]
Sep 17 00:20:10.662242 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:10.662211    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/438275c3-8268-4fb3-a4f2-0f8c75ed3b89-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-f5gwq\" (UID: \"438275c3-8268-4fb3-a4f2-0f8c75ed3b89\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-f5gwq"
Sep 17 00:20:10.662242 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:10.662248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/438275c3-8268-4fb3-a4f2-0f8c75ed3b89-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-f5gwq\" (UID: \"438275c3-8268-4fb3-a4f2-0f8c75ed3b89\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-f5gwq"
Sep 17 00:20:10.662500 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:10.662310    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgzm9\" (UniqueName: \"kubernetes.io/projected/438275c3-8268-4fb3-a4f2-0f8c75ed3b89-kube-api-access-dgzm9\") pod \"kuadrant-console-plugin-fbdd4f445-f5gwq\" (UID: \"438275c3-8268-4fb3-a4f2-0f8c75ed3b89\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-f5gwq"
Sep 17 00:20:10.763152 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:10.763107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/438275c3-8268-4fb3-a4f2-0f8c75ed3b89-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-f5gwq\" (UID: \"438275c3-8268-4fb3-a4f2-0f8c75ed3b89\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-f5gwq"
Sep 17 00:20:10.763152 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:10.763153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/438275c3-8268-4fb3-a4f2-0f8c75ed3b89-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-f5gwq\" (UID: \"438275c3-8268-4fb3-a4f2-0f8c75ed3b89\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-f5gwq"
Sep 17 00:20:10.763393 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:10.763201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgzm9\" (UniqueName: \"kubernetes.io/projected/438275c3-8268-4fb3-a4f2-0f8c75ed3b89-kube-api-access-dgzm9\") pod \"kuadrant-console-plugin-fbdd4f445-f5gwq\" (UID: \"438275c3-8268-4fb3-a4f2-0f8c75ed3b89\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-f5gwq"
Sep 17 00:20:10.763393 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:20:10.763266    2577 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 17 00:20:10.763393 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:20:10.763332    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/438275c3-8268-4fb3-a4f2-0f8c75ed3b89-plugin-serving-cert podName:438275c3-8268-4fb3-a4f2-0f8c75ed3b89 nodeName:}" failed. No retries permitted until 2026-09-17 00:20:11.263315861 +0000 UTC m=+624.121808974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/438275c3-8268-4fb3-a4f2-0f8c75ed3b89-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-f5gwq" (UID: "438275c3-8268-4fb3-a4f2-0f8c75ed3b89") : secret "plugin-serving-cert" not found
Sep 17 00:20:10.763966 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:10.763946    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/438275c3-8268-4fb3-a4f2-0f8c75ed3b89-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-f5gwq\" (UID: \"438275c3-8268-4fb3-a4f2-0f8c75ed3b89\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-f5gwq"
Sep 17 00:20:10.771709 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:10.771680    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgzm9\" (UniqueName: \"kubernetes.io/projected/438275c3-8268-4fb3-a4f2-0f8c75ed3b89-kube-api-access-dgzm9\") pod \"kuadrant-console-plugin-fbdd4f445-f5gwq\" (UID: \"438275c3-8268-4fb3-a4f2-0f8c75ed3b89\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-f5gwq"
Sep 17 00:20:11.038737 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:11.038709    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-kqwmq"
Sep 17 00:20:11.267171 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:11.267135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/438275c3-8268-4fb3-a4f2-0f8c75ed3b89-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-f5gwq\" (UID: \"438275c3-8268-4fb3-a4f2-0f8c75ed3b89\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-f5gwq"
Sep 17 00:20:11.269497 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:11.269469    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/438275c3-8268-4fb3-a4f2-0f8c75ed3b89-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-f5gwq\" (UID: \"438275c3-8268-4fb3-a4f2-0f8c75ed3b89\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-f5gwq"
Sep 17 00:20:11.499306 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:11.499275    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-f5gwq"
Sep 17 00:20:11.769282 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:11.769258    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-f5gwq"]
Sep 17 00:20:11.771008 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:20:11.770975    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod438275c3_8268_4fb3_a4f2_0f8c75ed3b89.slice/crio-e0bddf470e279a8ee93cd66741e10770ff8c3f2dfbf901b1d3bf38b5a9976b9c WatchSource:0}: Error finding container e0bddf470e279a8ee93cd66741e10770ff8c3f2dfbf901b1d3bf38b5a9976b9c: Status 404 returned error can't find the container with id e0bddf470e279a8ee93cd66741e10770ff8c3f2dfbf901b1d3bf38b5a9976b9c
Sep 17 00:20:12.105473 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:12.105386    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-f5gwq" event={"ID":"438275c3-8268-4fb3-a4f2-0f8c75ed3b89","Type":"ContainerStarted","Data":"e0bddf470e279a8ee93cd66741e10770ff8c3f2dfbf901b1d3bf38b5a9976b9c"}
Sep 17 00:20:13.058071 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:13.057476    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mxsms"
Sep 17 00:20:19.092114 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:19.092084    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn"
Sep 17 00:20:19.148878 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:19.148841    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-f5gwq" event={"ID":"438275c3-8268-4fb3-a4f2-0f8c75ed3b89","Type":"ContainerStarted","Data":"fa41d74f6f159f599add68452bde76021356def2306fde57b9fb84a2fd6900ad"}
Sep 17 00:20:19.173146 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:19.173063    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-f5gwq" podStartSLOduration=2.65254221 podStartE2EDuration="9.173041486s" totalImagesPullingTime="6.520499276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:20:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:20:19.172498152 +0000 UTC m=+632.030991287" watchObservedRunningTime="2026-09-17 00:20:19.173041486 +0000 UTC m=+632.031534619"
Sep 17 00:20:20.072472 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:20.072435    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h7s9x"]
Sep 17 00:20:20.077002 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:20.076984    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h7s9x"
Sep 17 00:20:20.088568 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:20.088545    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h7s9x"]
Sep 17 00:20:20.138781 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:20.138746    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fchdc\" (UniqueName: \"kubernetes.io/projected/31c26128-a014-49c1-a4cb-c9536ce57855-kube-api-access-fchdc\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-h7s9x\" (UID: \"31c26128-a014-49c1-a4cb-c9536ce57855\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h7s9x"
Sep 17 00:20:20.139137 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:20.138790    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31c26128-a014-49c1-a4cb-c9536ce57855-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-h7s9x\" (UID: \"31c26128-a014-49c1-a4cb-c9536ce57855\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h7s9x"
Sep 17 00:20:20.239808 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:20.239769    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fchdc\" (UniqueName: \"kubernetes.io/projected/31c26128-a014-49c1-a4cb-c9536ce57855-kube-api-access-fchdc\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-h7s9x\" (UID: \"31c26128-a014-49c1-a4cb-c9536ce57855\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h7s9x"
Sep 17 00:20:20.240007 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:20.239838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31c26128-a014-49c1-a4cb-c9536ce57855-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-h7s9x\" (UID: \"31c26128-a014-49c1-a4cb-c9536ce57855\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h7s9x"
Sep 17 00:20:20.240202 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:20.240181    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31c26128-a014-49c1-a4cb-c9536ce57855-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-h7s9x\" (UID: \"31c26128-a014-49c1-a4cb-c9536ce57855\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h7s9x"
Sep 17 00:20:20.248428 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:20.248391    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fchdc\" (UniqueName: \"kubernetes.io/projected/31c26128-a014-49c1-a4cb-c9536ce57855-kube-api-access-fchdc\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-h7s9x\" (UID: \"31c26128-a014-49c1-a4cb-c9536ce57855\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h7s9x"
Sep 17 00:20:20.387159 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:20.387071    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h7s9x"
Sep 17 00:20:20.667861 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:20.667832    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h7s9x"]
Sep 17 00:20:20.668515 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:20:20.668490    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c26128_a014_49c1_a4cb_c9536ce57855.slice/crio-e4f2174e0fdab53c6da8122b3f486400055570b5bfd2b71f447f3cd677c1bc03 WatchSource:0}: Error finding container e4f2174e0fdab53c6da8122b3f486400055570b5bfd2b71f447f3cd677c1bc03: Status 404 returned error can't find the container with id e4f2174e0fdab53c6da8122b3f486400055570b5bfd2b71f447f3cd677c1bc03
Sep 17 00:20:21.159871 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.159836    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h7s9x" event={"ID":"31c26128-a014-49c1-a4cb-c9536ce57855","Type":"ContainerStarted","Data":"325a000894024d7c210fca676bc9dbce1f408e2c70dcb977873dd954e6133354"}
Sep 17 00:20:21.160248 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.159888    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h7s9x" event={"ID":"31c26128-a014-49c1-a4cb-c9536ce57855","Type":"ContainerStarted","Data":"e4f2174e0fdab53c6da8122b3f486400055570b5bfd2b71f447f3cd677c1bc03"}
Sep 17 00:20:21.160248 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.159934    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h7s9x"
Sep 17 00:20:21.182990 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.182935    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h7s9x" podStartSLOduration=1.182918587 podStartE2EDuration="1.182918587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:20:21.180745576 +0000 UTC m=+634.039238711" watchObservedRunningTime="2026-09-17 00:20:21.182918587 +0000 UTC m=+634.041411722"
Sep 17 00:20:21.192732 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.192702    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h7s9x"]
Sep 17 00:20:21.202285 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.202260    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h7s9x"]
Sep 17 00:20:21.213867 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.213842    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn"]
Sep 17 00:20:21.214031 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.214010    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn" podUID="81252db6-9464-412c-8fd6-6bb866d67949" containerName="manager" containerID="cri-o://c2337c4c475de96e3fd36ec793c5f9aa03a020c2fd0930f13d27d05cb3b65cf9" gracePeriod=2
Sep 17 00:20:21.232676 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.232639    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gswjb"]
Sep 17 00:20:21.233688 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.233648    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31c26128-a014-49c1-a4cb-c9536ce57855" containerName="manager"
Sep 17 00:20:21.233688 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.233689    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="31c26128-a014-49c1-a4cb-c9536ce57855" containerName="manager"
Sep 17 00:20:21.233865 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.233780    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="31c26128-a014-49c1-a4cb-c9536ce57855" containerName="manager"
Sep 17 00:20:21.234764 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:20:21.234659    2577 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"81252db6-9464-412c-8fd6-6bb866d67949\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-9l4dn\": pods \"kuadrant-operator-controller-manager-76f8f686bf-9l4dn\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn"
Sep 17 00:20:21.237145 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.237126    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn"]
Sep 17 00:20:21.237234 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.237218    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gswjb"
Sep 17 00:20:21.238090 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.238067    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2p7xv"]
Sep 17 00:20:21.238309 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.238298    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81252db6-9464-412c-8fd6-6bb866d67949" containerName="manager"
Sep 17 00:20:21.238348 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.238311    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="81252db6-9464-412c-8fd6-6bb866d67949" containerName="manager"
Sep 17 00:20:21.238383 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.238363    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="81252db6-9464-412c-8fd6-6bb866d67949" containerName="manager"
Sep 17 00:20:21.239643 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:20:21.239617    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9l4dn\" is forbidden: User \"system:node:ip-10-0-139-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-27.ec2.internal' and this object" podUID="81252db6-9464-412c-8fd6-6bb866d67949" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn"
Sep 17 00:20:21.241148 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.241131    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2p7xv"
Sep 17 00:20:21.245851 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.245646    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gswjb"]
Sep 17 00:20:21.249200 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.249178    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mxsms"]
Sep 17 00:20:21.249524 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.249499    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mxsms" podUID="6cf42944-26fd-4299-99bc-3fd091be765f" containerName="manager" containerID="cri-o://56f4b155a45d22383d9b7d7774accaa84282d5e91c47d799038f99cb580c7731" gracePeriod=2
Sep 17 00:20:21.257055 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.257031    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2p7xv"]
Sep 17 00:20:21.261654 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.261631    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mxsms"]
Sep 17 00:20:21.267681 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:20:21.267653    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mxsms\" is forbidden: User \"system:node:ip-10-0-139-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-27.ec2.internal' and this object" podUID="6cf42944-26fd-4299-99bc-3fd091be765f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mxsms"
Sep 17 00:20:21.269999 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:20:21.269965    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9l4dn\" is forbidden: User \"system:node:ip-10-0-139-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-27.ec2.internal' and this object" podUID="81252db6-9464-412c-8fd6-6bb866d67949" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn"
Sep 17 00:20:21.273938 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.273916    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7p54z"]
Sep 17 00:20:21.274229 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.274216    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cf42944-26fd-4299-99bc-3fd091be765f" containerName="manager"
Sep 17 00:20:21.274276 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.274231    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cf42944-26fd-4299-99bc-3fd091be765f" containerName="manager"
Sep 17 00:20:21.274315 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.274289    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cf42944-26fd-4299-99bc-3fd091be765f" containerName="manager"
Sep 17 00:20:21.277164 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.277149    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7p54z"
Sep 17 00:20:21.294631 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.294567    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7p54z"]
Sep 17 00:20:21.297040 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:20:21.297007    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mxsms\" is forbidden: User \"system:node:ip-10-0-139-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-27.ec2.internal' and this object" podUID="6cf42944-26fd-4299-99bc-3fd091be765f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mxsms"
Sep 17 00:20:21.314600 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:20:21.314561    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9l4dn\" is forbidden: User \"system:node:ip-10-0-139-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-27.ec2.internal' and this object" podUID="81252db6-9464-412c-8fd6-6bb866d67949" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn"
Sep 17 00:20:21.349631 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.349594    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/894d5613-6157-4ba4-850b-6581b2b78ad5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-gswjb\" (UID: \"894d5613-6157-4ba4-850b-6581b2b78ad5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gswjb"
Sep 17 00:20:21.349812 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.349641    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bng7w\" (UniqueName: \"kubernetes.io/projected/c5c2d4fb-e8b4-478a-82b3-ade5fa66e810-kube-api-access-bng7w\") pod \"limitador-operator-controller-manager-74495c69f8-7p54z\" (UID: \"c5c2d4fb-e8b4-478a-82b3-ade5fa66e810\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7p54z"
Sep 17 00:20:21.349812 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.349669    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtbkg\" (UniqueName: \"kubernetes.io/projected/894d5613-6157-4ba4-850b-6581b2b78ad5-kube-api-access-wtbkg\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-gswjb\" (UID: \"894d5613-6157-4ba4-850b-6581b2b78ad5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gswjb"
Sep 17 00:20:21.349812 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.349778    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebf8d03a-13d3-459b-9f3f-778770e8a081-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2p7xv\" (UID: \"ebf8d03a-13d3-459b-9f3f-778770e8a081\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2p7xv"
Sep 17 00:20:21.349939 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.349820    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn84f\" (UniqueName: \"kubernetes.io/projected/ebf8d03a-13d3-459b-9f3f-778770e8a081-kube-api-access-xn84f\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2p7xv\" (UID: \"ebf8d03a-13d3-459b-9f3f-778770e8a081\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2p7xv"
Sep 17 00:20:21.451147 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.451110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtbkg\" (UniqueName: \"kubernetes.io/projected/894d5613-6157-4ba4-850b-6581b2b78ad5-kube-api-access-wtbkg\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-gswjb\" (UID: \"894d5613-6157-4ba4-850b-6581b2b78ad5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gswjb"
Sep 17 00:20:21.451277 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.451172    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebf8d03a-13d3-459b-9f3f-778770e8a081-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2p7xv\" (UID: \"ebf8d03a-13d3-459b-9f3f-778770e8a081\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2p7xv"
Sep 17 00:20:21.451277 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.451198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn84f\" (UniqueName: \"kubernetes.io/projected/ebf8d03a-13d3-459b-9f3f-778770e8a081-kube-api-access-xn84f\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2p7xv\" (UID: \"ebf8d03a-13d3-459b-9f3f-778770e8a081\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2p7xv"
Sep 17 00:20:21.451277 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.451267    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/894d5613-6157-4ba4-850b-6581b2b78ad5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-gswjb\" (UID: \"894d5613-6157-4ba4-850b-6581b2b78ad5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gswjb"
Sep 17 00:20:21.451449 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.451292    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bng7w\" (UniqueName: \"kubernetes.io/projected/c5c2d4fb-e8b4-478a-82b3-ade5fa66e810-kube-api-access-bng7w\") pod \"limitador-operator-controller-manager-74495c69f8-7p54z\" (UID: \"c5c2d4fb-e8b4-478a-82b3-ade5fa66e810\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7p54z"
Sep 17 00:20:21.451684 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.451653    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebf8d03a-13d3-459b-9f3f-778770e8a081-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2p7xv\" (UID: \"ebf8d03a-13d3-459b-9f3f-778770e8a081\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2p7xv"
Sep 17 00:20:21.451997 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.451971    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/894d5613-6157-4ba4-850b-6581b2b78ad5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-gswjb\" (UID: \"894d5613-6157-4ba4-850b-6581b2b78ad5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gswjb"
Sep 17 00:20:21.460210 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.460162    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bng7w\" (UniqueName: \"kubernetes.io/projected/c5c2d4fb-e8b4-478a-82b3-ade5fa66e810-kube-api-access-bng7w\") pod \"limitador-operator-controller-manager-74495c69f8-7p54z\" (UID: \"c5c2d4fb-e8b4-478a-82b3-ade5fa66e810\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7p54z"
Sep 17 00:20:21.460320 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.460242    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtbkg\" (UniqueName: \"kubernetes.io/projected/894d5613-6157-4ba4-850b-6581b2b78ad5-kube-api-access-wtbkg\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-gswjb\" (UID: \"894d5613-6157-4ba4-850b-6581b2b78ad5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gswjb"
Sep 17 00:20:21.461010 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.460989    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn84f\" (UniqueName: \"kubernetes.io/projected/ebf8d03a-13d3-459b-9f3f-778770e8a081-kube-api-access-xn84f\") pod \"kuadrant-operator-controller-manager-76f8f686bf-2p7xv\" (UID: \"ebf8d03a-13d3-459b-9f3f-778770e8a081\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2p7xv"
Sep 17 00:20:21.500068 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.500047    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn"
Sep 17 00:20:21.502093 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.502078    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mxsms"
Sep 17 00:20:21.502345 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:20:21.502326    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9l4dn\" is forbidden: User \"system:node:ip-10-0-139-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-27.ec2.internal' and this object" podUID="81252db6-9464-412c-8fd6-6bb866d67949" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn"
Sep 17 00:20:21.504244 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:20:21.504224    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mxsms\" is forbidden: User \"system:node:ip-10-0-139-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-27.ec2.internal' and this object" podUID="6cf42944-26fd-4299-99bc-3fd091be765f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mxsms"
Sep 17 00:20:21.506093 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:20:21.506073    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9l4dn\" is forbidden: User \"system:node:ip-10-0-139-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-27.ec2.internal' and this object" podUID="81252db6-9464-412c-8fd6-6bb866d67949" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn"
Sep 17 00:20:21.507939 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:20:21.507919    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mxsms\" is forbidden: User \"system:node:ip-10-0-139-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-27.ec2.internal' and this object" podUID="6cf42944-26fd-4299-99bc-3fd091be765f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mxsms"
Sep 17 00:20:21.552307 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.552281    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrw62\" (UniqueName: \"kubernetes.io/projected/6cf42944-26fd-4299-99bc-3fd091be765f-kube-api-access-vrw62\") pod \"6cf42944-26fd-4299-99bc-3fd091be765f\" (UID: \"6cf42944-26fd-4299-99bc-3fd091be765f\") "
Sep 17 00:20:21.552433 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.552341    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81252db6-9464-412c-8fd6-6bb866d67949-extensions-socket-volume\") pod \"81252db6-9464-412c-8fd6-6bb866d67949\" (UID: \"81252db6-9464-412c-8fd6-6bb866d67949\") "
Sep 17 00:20:21.552433 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.552401    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2qns\" (UniqueName: \"kubernetes.io/projected/81252db6-9464-412c-8fd6-6bb866d67949-kube-api-access-f2qns\") pod \"81252db6-9464-412c-8fd6-6bb866d67949\" (UID: \"81252db6-9464-412c-8fd6-6bb866d67949\") "
Sep 17 00:20:21.552915 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.552885    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81252db6-9464-412c-8fd6-6bb866d67949-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "81252db6-9464-412c-8fd6-6bb866d67949" (UID: "81252db6-9464-412c-8fd6-6bb866d67949"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:20:21.554522 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.554495    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81252db6-9464-412c-8fd6-6bb866d67949-kube-api-access-f2qns" (OuterVolumeSpecName: "kube-api-access-f2qns") pod "81252db6-9464-412c-8fd6-6bb866d67949" (UID: "81252db6-9464-412c-8fd6-6bb866d67949"). InnerVolumeSpecName "kube-api-access-f2qns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:20:21.554608 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.554557    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cf42944-26fd-4299-99bc-3fd091be765f-kube-api-access-vrw62" (OuterVolumeSpecName: "kube-api-access-vrw62") pod "6cf42944-26fd-4299-99bc-3fd091be765f" (UID: "6cf42944-26fd-4299-99bc-3fd091be765f"). InnerVolumeSpecName "kube-api-access-vrw62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:20:21.606985 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.606955    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cf42944-26fd-4299-99bc-3fd091be765f" path="/var/lib/kubelet/pods/6cf42944-26fd-4299-99bc-3fd091be765f/volumes"
Sep 17 00:20:21.607298 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.607286    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81252db6-9464-412c-8fd6-6bb866d67949" path="/var/lib/kubelet/pods/81252db6-9464-412c-8fd6-6bb866d67949/volumes"
Sep 17 00:20:21.653570 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.653542    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2qns\" (UniqueName: \"kubernetes.io/projected/81252db6-9464-412c-8fd6-6bb866d67949-kube-api-access-f2qns\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:20:21.653570 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.653569    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrw62\" (UniqueName: \"kubernetes.io/projected/6cf42944-26fd-4299-99bc-3fd091be765f-kube-api-access-vrw62\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:20:21.653570 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.653578    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81252db6-9464-412c-8fd6-6bb866d67949-extensions-socket-volume\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:20:21.666541 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.666518    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gswjb"
Sep 17 00:20:21.675503 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.675480    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2p7xv"
Sep 17 00:20:21.681540 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.681517    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7p54z"
Sep 17 00:20:21.951322 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.951292    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gswjb"]
Sep 17 00:20:21.952399 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:20:21.952372    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894d5613_6157_4ba4_850b_6581b2b78ad5.slice/crio-bda47836bebf79b31c65f032768bc87b51fce9673c5aaaf9a84fd755f051eb99 WatchSource:0}: Error finding container bda47836bebf79b31c65f032768bc87b51fce9673c5aaaf9a84fd755f051eb99: Status 404 returned error can't find the container with id bda47836bebf79b31c65f032768bc87b51fce9673c5aaaf9a84fd755f051eb99
Sep 17 00:20:21.968283 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.968259    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2p7xv"]
Sep 17 00:20:21.970729 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:20:21.970695    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf8d03a_13d3_459b_9f3f_778770e8a081.slice/crio-c3ebdf55b8bf45aee926fe97cdec4a53f843899cfcc3d45d205523f35386a0dd WatchSource:0}: Error finding container c3ebdf55b8bf45aee926fe97cdec4a53f843899cfcc3d45d205523f35386a0dd: Status 404 returned error can't find the container with id c3ebdf55b8bf45aee926fe97cdec4a53f843899cfcc3d45d205523f35386a0dd
Sep 17 00:20:21.971076 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:21.971041    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7p54z"]
Sep 17 00:20:21.973121 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:20:21.973088    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c2d4fb_e8b4_478a_82b3_ade5fa66e810.slice/crio-75b0047d7f811e2313ad3aa41ef72f0e8ea153fc638eea13e6a317ab76ecffab WatchSource:0}: Error finding container 75b0047d7f811e2313ad3aa41ef72f0e8ea153fc638eea13e6a317ab76ecffab: Status 404 returned error can't find the container with id 75b0047d7f811e2313ad3aa41ef72f0e8ea153fc638eea13e6a317ab76ecffab
Sep 17 00:20:22.166607 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:22.166574    2577 generic.go:356] "Generic (PLEG): container finished" podID="81252db6-9464-412c-8fd6-6bb866d67949" containerID="c2337c4c475de96e3fd36ec793c5f9aa03a020c2fd0930f13d27d05cb3b65cf9" exitCode=0
Sep 17 00:20:22.167081 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:22.166628    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9l4dn"
Sep 17 00:20:22.167081 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:22.166638    2577 scope.go:117] "RemoveContainer" containerID="c2337c4c475de96e3fd36ec793c5f9aa03a020c2fd0930f13d27d05cb3b65cf9"
Sep 17 00:20:22.169008 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:22.168981    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7p54z" event={"ID":"c5c2d4fb-e8b4-478a-82b3-ade5fa66e810","Type":"ContainerStarted","Data":"34392dfdf80f741c522fe6eaa0d5723f964da07822893aad51c9437c1c439956"}
Sep 17 00:20:22.169131 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:22.169042    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7p54z" event={"ID":"c5c2d4fb-e8b4-478a-82b3-ade5fa66e810","Type":"ContainerStarted","Data":"75b0047d7f811e2313ad3aa41ef72f0e8ea153fc638eea13e6a317ab76ecffab"}
Sep 17 00:20:22.169131 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:22.169075    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7p54z"
Sep 17 00:20:22.170910 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:22.170886    2577 generic.go:356] "Generic (PLEG): container finished" podID="6cf42944-26fd-4299-99bc-3fd091be765f" containerID="56f4b155a45d22383d9b7d7774accaa84282d5e91c47d799038f99cb580c7731" exitCode=0
Sep 17 00:20:22.171040 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:22.170929    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mxsms"
Sep 17 00:20:22.173205 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:22.173181    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2p7xv" event={"ID":"ebf8d03a-13d3-459b-9f3f-778770e8a081","Type":"ContainerStarted","Data":"0c2954ce18ad54985b78636c2ee7069068a687eb0a74f2ea4694cf71bcd52e9a"}
Sep 17 00:20:22.173291 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:22.173256    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2p7xv"
Sep 17 00:20:22.173291 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:22.173283    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2p7xv" event={"ID":"ebf8d03a-13d3-459b-9f3f-778770e8a081","Type":"ContainerStarted","Data":"c3ebdf55b8bf45aee926fe97cdec4a53f843899cfcc3d45d205523f35386a0dd"}
Sep 17 00:20:22.175652 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:22.175632    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gswjb" event={"ID":"894d5613-6157-4ba4-850b-6581b2b78ad5","Type":"ContainerStarted","Data":"9d8aa7101c5f2d43467abf0dd4f56709f1bc301c80f48de29640bcdeec9a9f52"}
Sep 17 00:20:22.175745 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:22.175673    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gswjb" event={"ID":"894d5613-6157-4ba4-850b-6581b2b78ad5","Type":"ContainerStarted","Data":"bda47836bebf79b31c65f032768bc87b51fce9673c5aaaf9a84fd755f051eb99"}
Sep 17 00:20:22.175848 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:20:22.175821    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h7s9x" container="manager"
Sep 17 00:20:22.180934 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:22.180913    2577 scope.go:117] "RemoveContainer" containerID="c2337c4c475de96e3fd36ec793c5f9aa03a020c2fd0930f13d27d05cb3b65cf9"
Sep 17 00:20:22.181731 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:20:22.181705    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2337c4c475de96e3fd36ec793c5f9aa03a020c2fd0930f13d27d05cb3b65cf9\": container with ID starting with c2337c4c475de96e3fd36ec793c5f9aa03a020c2fd0930f13d27d05cb3b65cf9 not found: ID does not exist" containerID="c2337c4c475de96e3fd36ec793c5f9aa03a020c2fd0930f13d27d05cb3b65cf9"
Sep 17 00:20:22.181849 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:22.181738    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2337c4c475de96e3fd36ec793c5f9aa03a020c2fd0930f13d27d05cb3b65cf9"} err="failed to get container status \"c2337c4c475de96e3fd36ec793c5f9aa03a020c2fd0930f13d27d05cb3b65cf9\": rpc error: code = NotFound desc = could not find container \"c2337c4c475de96e3fd36ec793c5f9aa03a020c2fd0930f13d27d05cb3b65cf9\": container with ID starting with c2337c4c475de96e3fd36ec793c5f9aa03a020c2fd0930f13d27d05cb3b65cf9 not found: ID does not exist"
Sep 17 00:20:22.181849 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:22.181759    2577 scope.go:117] "RemoveContainer" containerID="56f4b155a45d22383d9b7d7774accaa84282d5e91c47d799038f99cb580c7731"
Sep 17 00:20:22.189349 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:22.189332    2577 scope.go:117] "RemoveContainer" containerID="56f4b155a45d22383d9b7d7774accaa84282d5e91c47d799038f99cb580c7731"
Sep 17 00:20:22.189642 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:20:22.189622    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56f4b155a45d22383d9b7d7774accaa84282d5e91c47d799038f99cb580c7731\": container with ID starting with 56f4b155a45d22383d9b7d7774accaa84282d5e91c47d799038f99cb580c7731 not found: ID does not exist" containerID="56f4b155a45d22383d9b7d7774accaa84282d5e91c47d799038f99cb580c7731"
Sep 17 00:20:22.189691 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:22.189648    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56f4b155a45d22383d9b7d7774accaa84282d5e91c47d799038f99cb580c7731"} err="failed to get container status \"56f4b155a45d22383d9b7d7774accaa84282d5e91c47d799038f99cb580c7731\": rpc error: code = NotFound desc = could not find container \"56f4b155a45d22383d9b7d7774accaa84282d5e91c47d799038f99cb580c7731\": container with ID starting with 56f4b155a45d22383d9b7d7774accaa84282d5e91c47d799038f99cb580c7731 not found: ID does not exist"
Sep 17 00:20:22.203945 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:22.203872    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7p54z" podStartSLOduration=1.20385877 podStartE2EDuration="1.20385877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:20:22.201362456 +0000 UTC m=+635.059855591" watchObservedRunningTime="2026-09-17 00:20:22.20385877 +0000 UTC m=+635.062351905"
Sep 17 00:20:22.256033 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:22.255968    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2p7xv" podStartSLOduration=1.2559489990000001 podStartE2EDuration="1.255948999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:20:22.250631365 +0000 UTC m=+635.109124500" watchObservedRunningTime="2026-09-17 00:20:22.255948999 +0000 UTC m=+635.114442138"
Sep 17 00:20:23.184661 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:23.184623    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gswjb"
Sep 17 00:20:23.185169 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:23.184709    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h7s9x" podUID="31c26128-a014-49c1-a4cb-c9536ce57855" containerName="manager" containerID="cri-o://325a000894024d7c210fca676bc9dbce1f408e2c70dcb977873dd954e6133354" gracePeriod=2
Sep 17 00:20:23.415706 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:23.415682    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h7s9x"
Sep 17 00:20:23.417879 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:20:23.417851    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-h7s9x\" is forbidden: User \"system:node:ip-10-0-139-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-27.ec2.internal' and this object" podUID="31c26128-a014-49c1-a4cb-c9536ce57855" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h7s9x"
Sep 17 00:20:23.467467 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:23.467383    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31c26128-a014-49c1-a4cb-c9536ce57855-extensions-socket-volume\") pod \"31c26128-a014-49c1-a4cb-c9536ce57855\" (UID: \"31c26128-a014-49c1-a4cb-c9536ce57855\") "
Sep 17 00:20:23.467467 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:23.467444    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fchdc\" (UniqueName: \"kubernetes.io/projected/31c26128-a014-49c1-a4cb-c9536ce57855-kube-api-access-fchdc\") pod \"31c26128-a014-49c1-a4cb-c9536ce57855\" (UID: \"31c26128-a014-49c1-a4cb-c9536ce57855\") "
Sep 17 00:20:23.467758 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:23.467732    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c26128-a014-49c1-a4cb-c9536ce57855-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "31c26128-a014-49c1-a4cb-c9536ce57855" (UID: "31c26128-a014-49c1-a4cb-c9536ce57855"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:20:23.469607 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:23.469587    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c26128-a014-49c1-a4cb-c9536ce57855-kube-api-access-fchdc" (OuterVolumeSpecName: "kube-api-access-fchdc") pod "31c26128-a014-49c1-a4cb-c9536ce57855" (UID: "31c26128-a014-49c1-a4cb-c9536ce57855"). InnerVolumeSpecName "kube-api-access-fchdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:20:23.568260 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:23.568224    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fchdc\" (UniqueName: \"kubernetes.io/projected/31c26128-a014-49c1-a4cb-c9536ce57855-kube-api-access-fchdc\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:20:23.568260 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:23.568254    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31c26128-a014-49c1-a4cb-c9536ce57855-extensions-socket-volume\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:20:23.608315 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:23.608282    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31c26128-a014-49c1-a4cb-c9536ce57855" path="/var/lib/kubelet/pods/31c26128-a014-49c1-a4cb-c9536ce57855/volumes"
Sep 17 00:20:24.190718 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:24.190680    2577 generic.go:356] "Generic (PLEG): container finished" podID="31c26128-a014-49c1-a4cb-c9536ce57855" containerID="325a000894024d7c210fca676bc9dbce1f408e2c70dcb977873dd954e6133354" exitCode=0
Sep 17 00:20:24.190718 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:24.190723    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-h7s9x"
Sep 17 00:20:24.191214 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:24.190767    2577 scope.go:117] "RemoveContainer" containerID="325a000894024d7c210fca676bc9dbce1f408e2c70dcb977873dd954e6133354"
Sep 17 00:20:24.199047 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:24.199031    2577 scope.go:117] "RemoveContainer" containerID="325a000894024d7c210fca676bc9dbce1f408e2c70dcb977873dd954e6133354"
Sep 17 00:20:24.199323 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:20:24.199304    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"325a000894024d7c210fca676bc9dbce1f408e2c70dcb977873dd954e6133354\": container with ID starting with 325a000894024d7c210fca676bc9dbce1f408e2c70dcb977873dd954e6133354 not found: ID does not exist" containerID="325a000894024d7c210fca676bc9dbce1f408e2c70dcb977873dd954e6133354"
Sep 17 00:20:24.199380 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:24.199332    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325a000894024d7c210fca676bc9dbce1f408e2c70dcb977873dd954e6133354"} err="failed to get container status \"325a000894024d7c210fca676bc9dbce1f408e2c70dcb977873dd954e6133354\": rpc error: code = NotFound desc = could not find container \"325a000894024d7c210fca676bc9dbce1f408e2c70dcb977873dd954e6133354\": container with ID starting with 325a000894024d7c210fca676bc9dbce1f408e2c70dcb977873dd954e6133354 not found: ID does not exist"
Sep 17 00:20:33.186936 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:33.186904    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2p7xv"
Sep 17 00:20:33.187436 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:33.187006    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7p54z"
Sep 17 00:20:33.206484 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:33.206404    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gswjb" podStartSLOduration=12.206389103 podStartE2EDuration="12.206389103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:20:22.313959852 +0000 UTC m=+635.172452999" watchObservedRunningTime="2026-09-17 00:20:33.206389103 +0000 UTC m=+646.064882249"
Sep 17 00:20:34.193594 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:34.193560    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gswjb"
Sep 17 00:20:34.243543 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:34.243501    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2p7xv"]
Sep 17 00:20:34.243754 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:34.243731    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2p7xv" podUID="ebf8d03a-13d3-459b-9f3f-778770e8a081" containerName="manager" containerID="cri-o://0c2954ce18ad54985b78636c2ee7069068a687eb0a74f2ea4694cf71bcd52e9a" gracePeriod=10
Sep 17 00:20:34.492084 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:34.492058    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2p7xv"
Sep 17 00:20:34.512211 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:34.512175    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg"]
Sep 17 00:20:34.512523 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:34.512508    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebf8d03a-13d3-459b-9f3f-778770e8a081" containerName="manager"
Sep 17 00:20:34.512523 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:34.512523    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebf8d03a-13d3-459b-9f3f-778770e8a081" containerName="manager"
Sep 17 00:20:34.512620 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:34.512584    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebf8d03a-13d3-459b-9f3f-778770e8a081" containerName="manager"
Sep 17 00:20:34.514586 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:34.514559    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg"
Sep 17 00:20:34.533696 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:34.533665    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg"]
Sep 17 00:20:34.548294 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:34.548258    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebf8d03a-13d3-459b-9f3f-778770e8a081-extensions-socket-volume\") pod \"ebf8d03a-13d3-459b-9f3f-778770e8a081\" (UID: \"ebf8d03a-13d3-459b-9f3f-778770e8a081\") "
Sep 17 00:20:34.548479 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:34.548334    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xn84f\" (UniqueName: \"kubernetes.io/projected/ebf8d03a-13d3-459b-9f3f-778770e8a081-kube-api-access-xn84f\") pod \"ebf8d03a-13d3-459b-9f3f-778770e8a081\" (UID: \"ebf8d03a-13d3-459b-9f3f-778770e8a081\") "
Sep 17 00:20:34.548735 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:34.548709    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf8d03a-13d3-459b-9f3f-778770e8a081-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ebf8d03a-13d3-459b-9f3f-778770e8a081" (UID: "ebf8d03a-13d3-459b-9f3f-778770e8a081"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:20:34.550443 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:34.550406    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf8d03a-13d3-459b-9f3f-778770e8a081-kube-api-access-xn84f" (OuterVolumeSpecName: "kube-api-access-xn84f") pod "ebf8d03a-13d3-459b-9f3f-778770e8a081" (UID: "ebf8d03a-13d3-459b-9f3f-778770e8a081"). InnerVolumeSpecName "kube-api-access-xn84f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:20:34.649140 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:34.649104    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mtr5\" (UniqueName: \"kubernetes.io/projected/b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969-kube-api-access-5mtr5\") pod \"kuadrant-operator-controller-manager-5769f99d9f-64ngg\" (UID: \"b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg"
Sep 17 00:20:34.649313 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:34.649234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-64ngg\" (UID: \"b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg"
Sep 17 00:20:34.649313 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:34.649309    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn84f\" (UniqueName: \"kubernetes.io/projected/ebf8d03a-13d3-459b-9f3f-778770e8a081-kube-api-access-xn84f\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:20:34.649388 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:34.649326    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebf8d03a-13d3-459b-9f3f-778770e8a081-extensions-socket-volume\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:20:34.750274 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:34.750180    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mtr5\" (UniqueName: \"kubernetes.io/projected/b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969-kube-api-access-5mtr5\") pod \"kuadrant-operator-controller-manager-5769f99d9f-64ngg\" (UID: \"b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg"
Sep 17 00:20:34.750274 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:34.750258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-64ngg\" (UID: \"b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg"
Sep 17 00:20:34.750658 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:34.750639    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-64ngg\" (UID: \"b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg"
Sep 17 00:20:34.764827 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:34.764803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mtr5\" (UniqueName: \"kubernetes.io/projected/b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969-kube-api-access-5mtr5\") pod \"kuadrant-operator-controller-manager-5769f99d9f-64ngg\" (UID: \"b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg"
Sep 17 00:20:34.826508 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:34.826463    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg"
Sep 17 00:20:35.103945 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:35.103918    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg"]
Sep 17 00:20:35.106180 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:20:35.106152    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e90a8a_5e50_4d2f_8cbb_3ead9ca8b969.slice/crio-765d4a02261280ab5a36594f53ddfcd7b9190a0dbc33ed3c2b83bbed39fdb1dc WatchSource:0}: Error finding container 765d4a02261280ab5a36594f53ddfcd7b9190a0dbc33ed3c2b83bbed39fdb1dc: Status 404 returned error can't find the container with id 765d4a02261280ab5a36594f53ddfcd7b9190a0dbc33ed3c2b83bbed39fdb1dc
Sep 17 00:20:35.256543 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:35.256509    2577 generic.go:356] "Generic (PLEG): container finished" podID="ebf8d03a-13d3-459b-9f3f-778770e8a081" containerID="0c2954ce18ad54985b78636c2ee7069068a687eb0a74f2ea4694cf71bcd52e9a" exitCode=0
Sep 17 00:20:35.256998 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:35.256572    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2p7xv"
Sep 17 00:20:35.256998 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:35.256603    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2p7xv" event={"ID":"ebf8d03a-13d3-459b-9f3f-778770e8a081","Type":"ContainerDied","Data":"0c2954ce18ad54985b78636c2ee7069068a687eb0a74f2ea4694cf71bcd52e9a"}
Sep 17 00:20:35.256998 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:35.256699    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2p7xv" event={"ID":"ebf8d03a-13d3-459b-9f3f-778770e8a081","Type":"ContainerDied","Data":"c3ebdf55b8bf45aee926fe97cdec4a53f843899cfcc3d45d205523f35386a0dd"}
Sep 17 00:20:35.256998 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:35.256731    2577 scope.go:117] "RemoveContainer" containerID="0c2954ce18ad54985b78636c2ee7069068a687eb0a74f2ea4694cf71bcd52e9a"
Sep 17 00:20:35.258743 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:35.258718    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg" event={"ID":"b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969","Type":"ContainerStarted","Data":"2f0c6449ce251be029d06288ee0c364626ea75e57c1be1a3339920f930194397"}
Sep 17 00:20:35.258870 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:35.258767    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg" event={"ID":"b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969","Type":"ContainerStarted","Data":"765d4a02261280ab5a36594f53ddfcd7b9190a0dbc33ed3c2b83bbed39fdb1dc"}
Sep 17 00:20:35.258870 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:35.258791    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg"
Sep 17 00:20:35.270297 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:35.270274    2577 scope.go:117] "RemoveContainer" containerID="0c2954ce18ad54985b78636c2ee7069068a687eb0a74f2ea4694cf71bcd52e9a"
Sep 17 00:20:35.270626 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:20:35.270605    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c2954ce18ad54985b78636c2ee7069068a687eb0a74f2ea4694cf71bcd52e9a\": container with ID starting with 0c2954ce18ad54985b78636c2ee7069068a687eb0a74f2ea4694cf71bcd52e9a not found: ID does not exist" containerID="0c2954ce18ad54985b78636c2ee7069068a687eb0a74f2ea4694cf71bcd52e9a"
Sep 17 00:20:35.270700 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:35.270637    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2954ce18ad54985b78636c2ee7069068a687eb0a74f2ea4694cf71bcd52e9a"} err="failed to get container status \"0c2954ce18ad54985b78636c2ee7069068a687eb0a74f2ea4694cf71bcd52e9a\": rpc error: code = NotFound desc = could not find container \"0c2954ce18ad54985b78636c2ee7069068a687eb0a74f2ea4694cf71bcd52e9a\": container with ID starting with 0c2954ce18ad54985b78636c2ee7069068a687eb0a74f2ea4694cf71bcd52e9a not found: ID does not exist"
Sep 17 00:20:35.285888 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:35.285835    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg" podStartSLOduration=1.285819032 podStartE2EDuration="1.285819032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:20:35.283780218 +0000 UTC m=+648.142273356" watchObservedRunningTime="2026-09-17 00:20:35.285819032 +0000 UTC m=+648.144312167"
Sep 17 00:20:35.299670 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:35.299638    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2p7xv"]
Sep 17 00:20:35.305317 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:35.305291    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-2p7xv"]
Sep 17 00:20:35.606899 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:35.606868    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebf8d03a-13d3-459b-9f3f-778770e8a081" path="/var/lib/kubelet/pods/ebf8d03a-13d3-459b-9f3f-778770e8a081/volumes"
Sep 17 00:20:46.267932 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:46.267898    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg"
Sep 17 00:20:46.316825 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:46.316790    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gswjb"]
Sep 17 00:20:46.317065 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:46.317024    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gswjb" podUID="894d5613-6157-4ba4-850b-6581b2b78ad5" containerName="manager" containerID="cri-o://9d8aa7101c5f2d43467abf0dd4f56709f1bc301c80f48de29640bcdeec9a9f52" gracePeriod=10
Sep 17 00:20:46.557577 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:46.557552    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gswjb"
Sep 17 00:20:46.647627 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:46.647575    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/894d5613-6157-4ba4-850b-6581b2b78ad5-extensions-socket-volume\") pod \"894d5613-6157-4ba4-850b-6581b2b78ad5\" (UID: \"894d5613-6157-4ba4-850b-6581b2b78ad5\") "
Sep 17 00:20:46.647838 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:46.647661    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtbkg\" (UniqueName: \"kubernetes.io/projected/894d5613-6157-4ba4-850b-6581b2b78ad5-kube-api-access-wtbkg\") pod \"894d5613-6157-4ba4-850b-6581b2b78ad5\" (UID: \"894d5613-6157-4ba4-850b-6581b2b78ad5\") "
Sep 17 00:20:46.648058 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:46.648031    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894d5613-6157-4ba4-850b-6581b2b78ad5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "894d5613-6157-4ba4-850b-6581b2b78ad5" (UID: "894d5613-6157-4ba4-850b-6581b2b78ad5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:20:46.649882 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:46.649860    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/894d5613-6157-4ba4-850b-6581b2b78ad5-kube-api-access-wtbkg" (OuterVolumeSpecName: "kube-api-access-wtbkg") pod "894d5613-6157-4ba4-850b-6581b2b78ad5" (UID: "894d5613-6157-4ba4-850b-6581b2b78ad5"). InnerVolumeSpecName "kube-api-access-wtbkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:20:46.748324 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:46.748279    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtbkg\" (UniqueName: \"kubernetes.io/projected/894d5613-6157-4ba4-850b-6581b2b78ad5-kube-api-access-wtbkg\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:20:46.748324 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:46.748314    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/894d5613-6157-4ba4-850b-6581b2b78ad5-extensions-socket-volume\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:20:47.329795 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:47.329753    2577 generic.go:356] "Generic (PLEG): container finished" podID="894d5613-6157-4ba4-850b-6581b2b78ad5" containerID="9d8aa7101c5f2d43467abf0dd4f56709f1bc301c80f48de29640bcdeec9a9f52" exitCode=0
Sep 17 00:20:47.330199 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:47.329815    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gswjb"
Sep 17 00:20:47.330199 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:47.329836    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gswjb" event={"ID":"894d5613-6157-4ba4-850b-6581b2b78ad5","Type":"ContainerDied","Data":"9d8aa7101c5f2d43467abf0dd4f56709f1bc301c80f48de29640bcdeec9a9f52"}
Sep 17 00:20:47.330199 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:47.329915    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gswjb" event={"ID":"894d5613-6157-4ba4-850b-6581b2b78ad5","Type":"ContainerDied","Data":"bda47836bebf79b31c65f032768bc87b51fce9673c5aaaf9a84fd755f051eb99"}
Sep 17 00:20:47.330199 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:47.329939    2577 scope.go:117] "RemoveContainer" containerID="9d8aa7101c5f2d43467abf0dd4f56709f1bc301c80f48de29640bcdeec9a9f52"
Sep 17 00:20:47.339741 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:47.339723    2577 scope.go:117] "RemoveContainer" containerID="9d8aa7101c5f2d43467abf0dd4f56709f1bc301c80f48de29640bcdeec9a9f52"
Sep 17 00:20:47.340059 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:20:47.340035    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d8aa7101c5f2d43467abf0dd4f56709f1bc301c80f48de29640bcdeec9a9f52\": container with ID starting with 9d8aa7101c5f2d43467abf0dd4f56709f1bc301c80f48de29640bcdeec9a9f52 not found: ID does not exist" containerID="9d8aa7101c5f2d43467abf0dd4f56709f1bc301c80f48de29640bcdeec9a9f52"
Sep 17 00:20:47.340114 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:47.340068    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d8aa7101c5f2d43467abf0dd4f56709f1bc301c80f48de29640bcdeec9a9f52"} err="failed to get container status \"9d8aa7101c5f2d43467abf0dd4f56709f1bc301c80f48de29640bcdeec9a9f52\": rpc error: code = NotFound desc = could not find container \"9d8aa7101c5f2d43467abf0dd4f56709f1bc301c80f48de29640bcdeec9a9f52\": container with ID starting with 9d8aa7101c5f2d43467abf0dd4f56709f1bc301c80f48de29640bcdeec9a9f52 not found: ID does not exist"
Sep 17 00:20:47.352796 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:47.352766    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gswjb"]
Sep 17 00:20:47.359775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:47.359746    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-gswjb"]
Sep 17 00:20:47.606811 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:20:47.606777    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="894d5613-6157-4ba4-850b-6581b2b78ad5" path="/var/lib/kubelet/pods/894d5613-6157-4ba4-850b-6581b2b78ad5/volumes"
Sep 17 00:21:06.294240 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:21:06.294201    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:21:06.294921 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:21:06.294519    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="894d5613-6157-4ba4-850b-6581b2b78ad5" containerName="manager"
Sep 17 00:21:06.294921 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:21:06.294531    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="894d5613-6157-4ba4-850b-6581b2b78ad5" containerName="manager"
Sep 17 00:21:06.294921 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:21:06.294581    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="894d5613-6157-4ba4-850b-6581b2b78ad5" containerName="manager"
Sep 17 00:21:06.320316 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:21:06.320280    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:21:06.320512 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:21:06.320385    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"
Sep 17 00:21:06.322887 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:21:06.322861    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 00:21:06.349471 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:21:06.349442    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:21:06.501381 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:21:06.501344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d4d3ebb4-cb8f-4a46-a9b7-deef9a612af2-config-file\") pod \"limitador-limitador-84994f7ffc-r7x6w\" (UID: \"d4d3ebb4-cb8f-4a46-a9b7-deef9a612af2\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"
Sep 17 00:21:06.501381 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:21:06.501389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2glhb\" (UniqueName: \"kubernetes.io/projected/d4d3ebb4-cb8f-4a46-a9b7-deef9a612af2-kube-api-access-2glhb\") pod \"limitador-limitador-84994f7ffc-r7x6w\" (UID: \"d4d3ebb4-cb8f-4a46-a9b7-deef9a612af2\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"
Sep 17 00:21:06.601864 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:21:06.601769    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d4d3ebb4-cb8f-4a46-a9b7-deef9a612af2-config-file\") pod \"limitador-limitador-84994f7ffc-r7x6w\" (UID: \"d4d3ebb4-cb8f-4a46-a9b7-deef9a612af2\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"
Sep 17 00:21:06.601864 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:21:06.601828    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2glhb\" (UniqueName: \"kubernetes.io/projected/d4d3ebb4-cb8f-4a46-a9b7-deef9a612af2-kube-api-access-2glhb\") pod \"limitador-limitador-84994f7ffc-r7x6w\" (UID: \"d4d3ebb4-cb8f-4a46-a9b7-deef9a612af2\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"
Sep 17 00:21:06.602380 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:21:06.602358    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d4d3ebb4-cb8f-4a46-a9b7-deef9a612af2-config-file\") pod \"limitador-limitador-84994f7ffc-r7x6w\" (UID: \"d4d3ebb4-cb8f-4a46-a9b7-deef9a612af2\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"
Sep 17 00:21:06.609327 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:21:06.609294    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2glhb\" (UniqueName: \"kubernetes.io/projected/d4d3ebb4-cb8f-4a46-a9b7-deef9a612af2-kube-api-access-2glhb\") pod \"limitador-limitador-84994f7ffc-r7x6w\" (UID: \"d4d3ebb4-cb8f-4a46-a9b7-deef9a612af2\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"
Sep 17 00:21:06.631072 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:21:06.631042    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"
Sep 17 00:21:06.902340 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:21:06.902311    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:21:06.904874 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:21:06.904839    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d3ebb4_cb8f_4a46_a9b7_deef9a612af2.slice/crio-eb7a2484381f2f3a1a8bcd13ccb4ce1cd864b8ef79b5546e85462456aa13e1b3 WatchSource:0}: Error finding container eb7a2484381f2f3a1a8bcd13ccb4ce1cd864b8ef79b5546e85462456aa13e1b3: Status 404 returned error can't find the container with id eb7a2484381f2f3a1a8bcd13ccb4ce1cd864b8ef79b5546e85462456aa13e1b3
Sep 17 00:21:06.907271 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:21:06.907255    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 00:21:07.429375 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:21:07.429339    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-r7x6w" event={"ID":"d4d3ebb4-cb8f-4a46-a9b7-deef9a612af2","Type":"ContainerStarted","Data":"eb7a2484381f2f3a1a8bcd13ccb4ce1cd864b8ef79b5546e85462456aa13e1b3"}
Sep 17 00:21:10.450430 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:21:10.450366    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-r7x6w" event={"ID":"d4d3ebb4-cb8f-4a46-a9b7-deef9a612af2","Type":"ContainerStarted","Data":"93cfeb94381e66775cb89d4c0e492b99de6d3221b269369c51d66118b9f3cfad"}
Sep 17 00:21:10.450886 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:21:10.450590    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"
Sep 17 00:21:10.466241 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:21:10.466190    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-r7x6w" podStartSLOduration=1.778803317 podStartE2EDuration="4.46617632s" totalImagesPullingTime="2.687373003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:21:10.464255337 +0000 UTC m=+683.322748468" watchObservedRunningTime="2026-09-17 00:21:10.46617632 +0000 UTC m=+683.324669455"
Sep 17 00:21:21.456563 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:21:21.456530    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"
Sep 17 00:22:21.404857 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:21.404812    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:22:37.607276 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:37.607242    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:22:50.996593 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:50.996552    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"]
Sep 17 00:22:51.000060 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.000037    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"
Sep 17 00:22:51.003664 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.003636    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 17 00:22:51.003774 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.003743    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jfhbm\""
Sep 17 00:22:51.003839 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.003749    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 17 00:22:51.003898 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.003843    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 17 00:22:51.009112 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.009077    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"]
Sep 17 00:22:51.194605 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.194562    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5ccd1e5-3fb1-41f4-a707-28de7a34b07e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kzq6k\" (UID: \"b5ccd1e5-3fb1-41f4-a707-28de7a34b07e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"
Sep 17 00:22:51.194605 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.194607    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2gkw\" (UniqueName: \"kubernetes.io/projected/b5ccd1e5-3fb1-41f4-a707-28de7a34b07e-kube-api-access-v2gkw\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kzq6k\" (UID: \"b5ccd1e5-3fb1-41f4-a707-28de7a34b07e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"
Sep 17 00:22:51.194833 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.194627    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5ccd1e5-3fb1-41f4-a707-28de7a34b07e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kzq6k\" (UID: \"b5ccd1e5-3fb1-41f4-a707-28de7a34b07e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"
Sep 17 00:22:51.194833 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.194722    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5ccd1e5-3fb1-41f4-a707-28de7a34b07e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kzq6k\" (UID: \"b5ccd1e5-3fb1-41f4-a707-28de7a34b07e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"
Sep 17 00:22:51.194833 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.194792    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5ccd1e5-3fb1-41f4-a707-28de7a34b07e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kzq6k\" (UID: \"b5ccd1e5-3fb1-41f4-a707-28de7a34b07e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"
Sep 17 00:22:51.194833 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.194827    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5ccd1e5-3fb1-41f4-a707-28de7a34b07e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kzq6k\" (UID: \"b5ccd1e5-3fb1-41f4-a707-28de7a34b07e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"
Sep 17 00:22:51.295785 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.295689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5ccd1e5-3fb1-41f4-a707-28de7a34b07e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kzq6k\" (UID: \"b5ccd1e5-3fb1-41f4-a707-28de7a34b07e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"
Sep 17 00:22:51.295785 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.295734    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2gkw\" (UniqueName: \"kubernetes.io/projected/b5ccd1e5-3fb1-41f4-a707-28de7a34b07e-kube-api-access-v2gkw\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kzq6k\" (UID: \"b5ccd1e5-3fb1-41f4-a707-28de7a34b07e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"
Sep 17 00:22:51.296013 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.295823    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5ccd1e5-3fb1-41f4-a707-28de7a34b07e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kzq6k\" (UID: \"b5ccd1e5-3fb1-41f4-a707-28de7a34b07e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"
Sep 17 00:22:51.296013 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.295898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5ccd1e5-3fb1-41f4-a707-28de7a34b07e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kzq6k\" (UID: \"b5ccd1e5-3fb1-41f4-a707-28de7a34b07e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"
Sep 17 00:22:51.296013 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.295957    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5ccd1e5-3fb1-41f4-a707-28de7a34b07e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kzq6k\" (UID: \"b5ccd1e5-3fb1-41f4-a707-28de7a34b07e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"
Sep 17 00:22:51.296013 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.295994    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5ccd1e5-3fb1-41f4-a707-28de7a34b07e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kzq6k\" (UID: \"b5ccd1e5-3fb1-41f4-a707-28de7a34b07e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"
Sep 17 00:22:51.296349 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.296317    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5ccd1e5-3fb1-41f4-a707-28de7a34b07e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kzq6k\" (UID: \"b5ccd1e5-3fb1-41f4-a707-28de7a34b07e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"
Sep 17 00:22:51.296489 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.296316    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5ccd1e5-3fb1-41f4-a707-28de7a34b07e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kzq6k\" (UID: \"b5ccd1e5-3fb1-41f4-a707-28de7a34b07e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"
Sep 17 00:22:51.296489 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.296378    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5ccd1e5-3fb1-41f4-a707-28de7a34b07e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kzq6k\" (UID: \"b5ccd1e5-3fb1-41f4-a707-28de7a34b07e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"
Sep 17 00:22:51.298101 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.298079    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5ccd1e5-3fb1-41f4-a707-28de7a34b07e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kzq6k\" (UID: \"b5ccd1e5-3fb1-41f4-a707-28de7a34b07e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"
Sep 17 00:22:51.298513 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.298493    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5ccd1e5-3fb1-41f4-a707-28de7a34b07e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kzq6k\" (UID: \"b5ccd1e5-3fb1-41f4-a707-28de7a34b07e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"
Sep 17 00:22:51.302901 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.302876    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2gkw\" (UniqueName: \"kubernetes.io/projected/b5ccd1e5-3fb1-41f4-a707-28de7a34b07e-kube-api-access-v2gkw\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kzq6k\" (UID: \"b5ccd1e5-3fb1-41f4-a707-28de7a34b07e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"
Sep 17 00:22:51.310556 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.310535    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"
Sep 17 00:22:51.585320 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.585289    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"]
Sep 17 00:22:51.587090 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:22:51.587059    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ccd1e5_3fb1_41f4_a707_28de7a34b07e.slice/crio-aa098bc2951aa3e17bb72da2199804709819fcae26817abeddeb1518140ec6bc WatchSource:0}: Error finding container aa098bc2951aa3e17bb72da2199804709819fcae26817abeddeb1518140ec6bc: Status 404 returned error can't find the container with id aa098bc2951aa3e17bb72da2199804709819fcae26817abeddeb1518140ec6bc
Sep 17 00:22:51.703023 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.702984    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:22:51.960225 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:51.960182    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k" event={"ID":"b5ccd1e5-3fb1-41f4-a707-28de7a34b07e","Type":"ContainerStarted","Data":"aa098bc2951aa3e17bb72da2199804709819fcae26817abeddeb1518140ec6bc"}
Sep 17 00:22:52.966522 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:52.966483    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k" event={"ID":"b5ccd1e5-3fb1-41f4-a707-28de7a34b07e","Type":"ContainerStarted","Data":"392ec751d0047f90293bebd6c33555487f14e8e7a8026adc186e58534f376840"}
Sep 17 00:22:52.982912 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:22:52.982862    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k" podStartSLOduration=1.993206491 podStartE2EDuration="2.982846581s" totalImagesPullingTime="989.64009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:22:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:22:52.982035584 +0000 UTC m=+785.840528741" watchObservedRunningTime="2026-09-17 00:22:52.982846581 +0000 UTC m=+785.841339716"
Sep 17 00:23:01.311014 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:01.310955    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"
Sep 17 00:23:01.311631 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:01.311047    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"
Sep 17 00:23:01.323694 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:01.323670    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"
Sep 17 00:23:02.025797 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:02.025767    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kzq6k"
Sep 17 00:23:12.201992 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:12.201905    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:23:24.499075 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:24.499037    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"]
Sep 17 00:23:24.503994 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:24.503971    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"
Sep 17 00:23:24.506483 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:24.506456    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 17 00:23:24.512427 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:24.512381    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"]
Sep 17 00:23:24.559997 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:24.559956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48lk2\" (UniqueName: \"kubernetes.io/projected/4625e67b-a3ce-4f58-86b6-c0224538815f-kube-api-access-48lk2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-728fr\" (UID: \"4625e67b-a3ce-4f58-86b6-c0224538815f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"
Sep 17 00:23:24.559997 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:24.560000    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4625e67b-a3ce-4f58-86b6-c0224538815f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-728fr\" (UID: \"4625e67b-a3ce-4f58-86b6-c0224538815f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"
Sep 17 00:23:24.560213 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:24.560035    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4625e67b-a3ce-4f58-86b6-c0224538815f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-728fr\" (UID: \"4625e67b-a3ce-4f58-86b6-c0224538815f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"
Sep 17 00:23:24.560213 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:24.560100    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4625e67b-a3ce-4f58-86b6-c0224538815f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-728fr\" (UID: \"4625e67b-a3ce-4f58-86b6-c0224538815f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"
Sep 17 00:23:24.560213 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:24.560142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4625e67b-a3ce-4f58-86b6-c0224538815f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-728fr\" (UID: \"4625e67b-a3ce-4f58-86b6-c0224538815f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"
Sep 17 00:23:24.560213 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:24.560171    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4625e67b-a3ce-4f58-86b6-c0224538815f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-728fr\" (UID: \"4625e67b-a3ce-4f58-86b6-c0224538815f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"
Sep 17 00:23:24.660727 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:24.660685    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4625e67b-a3ce-4f58-86b6-c0224538815f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-728fr\" (UID: \"4625e67b-a3ce-4f58-86b6-c0224538815f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"
Sep 17 00:23:24.660727 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:24.660736    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4625e67b-a3ce-4f58-86b6-c0224538815f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-728fr\" (UID: \"4625e67b-a3ce-4f58-86b6-c0224538815f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"
Sep 17 00:23:24.660974 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:24.660760    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4625e67b-a3ce-4f58-86b6-c0224538815f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-728fr\" (UID: \"4625e67b-a3ce-4f58-86b6-c0224538815f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"
Sep 17 00:23:24.660974 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:24.660785    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4625e67b-a3ce-4f58-86b6-c0224538815f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-728fr\" (UID: \"4625e67b-a3ce-4f58-86b6-c0224538815f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"
Sep 17 00:23:24.660974 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:24.660815    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48lk2\" (UniqueName: \"kubernetes.io/projected/4625e67b-a3ce-4f58-86b6-c0224538815f-kube-api-access-48lk2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-728fr\" (UID: \"4625e67b-a3ce-4f58-86b6-c0224538815f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"
Sep 17 00:23:24.660974 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:24.660841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4625e67b-a3ce-4f58-86b6-c0224538815f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-728fr\" (UID: \"4625e67b-a3ce-4f58-86b6-c0224538815f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"
Sep 17 00:23:24.661217 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:24.661185    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4625e67b-a3ce-4f58-86b6-c0224538815f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-728fr\" (UID: \"4625e67b-a3ce-4f58-86b6-c0224538815f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"
Sep 17 00:23:24.661289 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:24.661174    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4625e67b-a3ce-4f58-86b6-c0224538815f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-728fr\" (UID: \"4625e67b-a3ce-4f58-86b6-c0224538815f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"
Sep 17 00:23:24.661289 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:24.661255    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4625e67b-a3ce-4f58-86b6-c0224538815f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-728fr\" (UID: \"4625e67b-a3ce-4f58-86b6-c0224538815f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"
Sep 17 00:23:24.663208 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:24.663185    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4625e67b-a3ce-4f58-86b6-c0224538815f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-728fr\" (UID: \"4625e67b-a3ce-4f58-86b6-c0224538815f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"
Sep 17 00:23:24.663379 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:24.663361    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4625e67b-a3ce-4f58-86b6-c0224538815f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-728fr\" (UID: \"4625e67b-a3ce-4f58-86b6-c0224538815f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"
Sep 17 00:23:24.674463 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:24.674434    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48lk2\" (UniqueName: \"kubernetes.io/projected/4625e67b-a3ce-4f58-86b6-c0224538815f-kube-api-access-48lk2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-728fr\" (UID: \"4625e67b-a3ce-4f58-86b6-c0224538815f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"
Sep 17 00:23:24.815601 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:24.815505    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"
Sep 17 00:23:25.088968 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:25.088882    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"]
Sep 17 00:23:25.092496 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:23:25.092466    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4625e67b_a3ce_4f58_86b6_c0224538815f.slice/crio-0418378a8759e2ad6f31d23d7c4c99f39c3d160922e8d8fa618863f1e59c7457 WatchSource:0}: Error finding container 0418378a8759e2ad6f31d23d7c4c99f39c3d160922e8d8fa618863f1e59c7457: Status 404 returned error can't find the container with id 0418378a8759e2ad6f31d23d7c4c99f39c3d160922e8d8fa618863f1e59c7457
Sep 17 00:23:25.128518 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:25.128483    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr" event={"ID":"4625e67b-a3ce-4f58-86b6-c0224538815f","Type":"ContainerStarted","Data":"0418378a8759e2ad6f31d23d7c4c99f39c3d160922e8d8fa618863f1e59c7457"}
Sep 17 00:23:25.802003 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:25.801963    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:23:26.135645 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:26.135549    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr" event={"ID":"4625e67b-a3ce-4f58-86b6-c0224538815f","Type":"ContainerStarted","Data":"c1ae0deeafb08598e6e2ccf63401a98913c7df3b4e3c7ebe6025a80e6b4f2a3f"}
Sep 17 00:23:26.151916 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:26.151825    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr" podStartSLOduration=1.968451889 podStartE2EDuration="2.151801224s" totalImagesPullingTime="183.349335ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:23:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:23:26.151364654 +0000 UTC m=+819.009857790" watchObservedRunningTime="2026-09-17 00:23:26.151801224 +0000 UTC m=+819.010294362"
Sep 17 00:23:34.091459 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:34.091409    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:23:34.816116 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:34.816079    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"
Sep 17 00:23:34.816321 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:34.816165    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"
Sep 17 00:23:34.828615 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:34.828590    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"
Sep 17 00:23:35.192612 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:35.192584    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-728fr"
Sep 17 00:23:39.990618 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:39.990258    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"]
Sep 17 00:23:39.993978 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:39.993956    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"
Sep 17 00:23:39.996373 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:39.996348    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 17 00:23:40.003397 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:40.003375    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"]
Sep 17 00:23:40.097099 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:40.097063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1d34169-b614-4582-954a-8f3bd7f72819-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bdhcj\" (UID: \"d1d34169-b614-4582-954a-8f3bd7f72819\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"
Sep 17 00:23:40.097099 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:40.097099    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1d34169-b614-4582-954a-8f3bd7f72819-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bdhcj\" (UID: \"d1d34169-b614-4582-954a-8f3bd7f72819\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"
Sep 17 00:23:40.097305 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:40.097125    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6l6b\" (UniqueName: \"kubernetes.io/projected/d1d34169-b614-4582-954a-8f3bd7f72819-kube-api-access-b6l6b\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bdhcj\" (UID: \"d1d34169-b614-4582-954a-8f3bd7f72819\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"
Sep 17 00:23:40.097305 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:40.097195    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1d34169-b614-4582-954a-8f3bd7f72819-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bdhcj\" (UID: \"d1d34169-b614-4582-954a-8f3bd7f72819\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"
Sep 17 00:23:40.097305 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:40.097237    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1d34169-b614-4582-954a-8f3bd7f72819-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bdhcj\" (UID: \"d1d34169-b614-4582-954a-8f3bd7f72819\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"
Sep 17 00:23:40.097305 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:40.097258    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1d34169-b614-4582-954a-8f3bd7f72819-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bdhcj\" (UID: \"d1d34169-b614-4582-954a-8f3bd7f72819\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"
Sep 17 00:23:40.198672 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:40.198636    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1d34169-b614-4582-954a-8f3bd7f72819-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bdhcj\" (UID: \"d1d34169-b614-4582-954a-8f3bd7f72819\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"
Sep 17 00:23:40.198672 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:40.198672    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1d34169-b614-4582-954a-8f3bd7f72819-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bdhcj\" (UID: \"d1d34169-b614-4582-954a-8f3bd7f72819\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"
Sep 17 00:23:40.198911 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:40.198693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6l6b\" (UniqueName: \"kubernetes.io/projected/d1d34169-b614-4582-954a-8f3bd7f72819-kube-api-access-b6l6b\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bdhcj\" (UID: \"d1d34169-b614-4582-954a-8f3bd7f72819\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"
Sep 17 00:23:40.198911 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:40.198716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1d34169-b614-4582-954a-8f3bd7f72819-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bdhcj\" (UID: \"d1d34169-b614-4582-954a-8f3bd7f72819\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"
Sep 17 00:23:40.198911 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:40.198736    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1d34169-b614-4582-954a-8f3bd7f72819-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bdhcj\" (UID: \"d1d34169-b614-4582-954a-8f3bd7f72819\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"
Sep 17 00:23:40.198911 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:40.198757    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1d34169-b614-4582-954a-8f3bd7f72819-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bdhcj\" (UID: \"d1d34169-b614-4582-954a-8f3bd7f72819\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"
Sep 17 00:23:40.199110 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:40.199089    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1d34169-b614-4582-954a-8f3bd7f72819-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bdhcj\" (UID: \"d1d34169-b614-4582-954a-8f3bd7f72819\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"
Sep 17 00:23:40.199155 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:40.199111    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1d34169-b614-4582-954a-8f3bd7f72819-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bdhcj\" (UID: \"d1d34169-b614-4582-954a-8f3bd7f72819\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"
Sep 17 00:23:40.199192 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:40.199160    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1d34169-b614-4582-954a-8f3bd7f72819-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bdhcj\" (UID: \"d1d34169-b614-4582-954a-8f3bd7f72819\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"
Sep 17 00:23:40.200982 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:40.200964    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1d34169-b614-4582-954a-8f3bd7f72819-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bdhcj\" (UID: \"d1d34169-b614-4582-954a-8f3bd7f72819\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"
Sep 17 00:23:40.201106 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:40.201090    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1d34169-b614-4582-954a-8f3bd7f72819-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bdhcj\" (UID: \"d1d34169-b614-4582-954a-8f3bd7f72819\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"
Sep 17 00:23:40.205985 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:40.205962    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6l6b\" (UniqueName: \"kubernetes.io/projected/d1d34169-b614-4582-954a-8f3bd7f72819-kube-api-access-b6l6b\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bdhcj\" (UID: \"d1d34169-b614-4582-954a-8f3bd7f72819\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"
Sep 17 00:23:40.305988 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:40.305889    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"
Sep 17 00:23:40.576778 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:40.576695    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"]
Sep 17 00:23:40.580212 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:23:40.580183    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d34169_b614_4582_954a_8f3bd7f72819.slice/crio-b654bd0dc4e935e9a3acb11f21f8df9bb4112792a1b3196544b0fa2707ab7452 WatchSource:0}: Error finding container b654bd0dc4e935e9a3acb11f21f8df9bb4112792a1b3196544b0fa2707ab7452: Status 404 returned error can't find the container with id b654bd0dc4e935e9a3acb11f21f8df9bb4112792a1b3196544b0fa2707ab7452
Sep 17 00:23:41.215454 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:41.215422    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj" event={"ID":"d1d34169-b614-4582-954a-8f3bd7f72819","Type":"ContainerStarted","Data":"ed750979d9723de2579a0f5edb0bbc4619e0e0ac971b91f9b5abe01d98758a0d"}
Sep 17 00:23:41.215803 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:41.215480    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj" event={"ID":"d1d34169-b614-4582-954a-8f3bd7f72819","Type":"ContainerStarted","Data":"b654bd0dc4e935e9a3acb11f21f8df9bb4112792a1b3196544b0fa2707ab7452"}
Sep 17 00:23:41.231408 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:41.231366    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj" podStartSLOduration=2.011529567 podStartE2EDuration="2.2313329s" totalImagesPullingTime="219.803333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:23:41.230542381 +0000 UTC m=+834.089035517" watchObservedRunningTime="2026-09-17 00:23:41.2313329 +0000 UTC m=+834.089826017"
Sep 17 00:23:41.804266 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:41.804230    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:23:50.306669 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:50.306628    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"
Sep 17 00:23:50.307081 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:50.306697    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"
Sep 17 00:23:50.319124 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:50.319095    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"
Sep 17 00:23:51.276664 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:51.276631    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bdhcj"
Sep 17 00:23:52.498405 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:52.498369    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"]
Sep 17 00:23:52.501996 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:52.501975    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"
Sep 17 00:23:52.504591 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:52.504571    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 17 00:23:52.510141 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:52.510118    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"]
Sep 17 00:23:52.603067 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:52.603026    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176b5107-4763-404b-b51b-39bd741a6835-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2\" (UID: \"176b5107-4763-404b-b51b-39bd741a6835\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"
Sep 17 00:23:52.603240 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:52.603078    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176b5107-4763-404b-b51b-39bd741a6835-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2\" (UID: \"176b5107-4763-404b-b51b-39bd741a6835\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"
Sep 17 00:23:52.603240 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:52.603145    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176b5107-4763-404b-b51b-39bd741a6835-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2\" (UID: \"176b5107-4763-404b-b51b-39bd741a6835\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"
Sep 17 00:23:52.603240 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:52.603189    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77pbk\" (UniqueName: \"kubernetes.io/projected/176b5107-4763-404b-b51b-39bd741a6835-kube-api-access-77pbk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2\" (UID: \"176b5107-4763-404b-b51b-39bd741a6835\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"
Sep 17 00:23:52.603240 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:52.603237    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176b5107-4763-404b-b51b-39bd741a6835-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2\" (UID: \"176b5107-4763-404b-b51b-39bd741a6835\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"
Sep 17 00:23:52.603372 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:52.603268    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176b5107-4763-404b-b51b-39bd741a6835-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2\" (UID: \"176b5107-4763-404b-b51b-39bd741a6835\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"
Sep 17 00:23:52.703792 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:52.703756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176b5107-4763-404b-b51b-39bd741a6835-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2\" (UID: \"176b5107-4763-404b-b51b-39bd741a6835\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"
Sep 17 00:23:52.703963 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:52.703813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176b5107-4763-404b-b51b-39bd741a6835-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2\" (UID: \"176b5107-4763-404b-b51b-39bd741a6835\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"
Sep 17 00:23:52.703963 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:52.703840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176b5107-4763-404b-b51b-39bd741a6835-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2\" (UID: \"176b5107-4763-404b-b51b-39bd741a6835\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"
Sep 17 00:23:52.703963 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:52.703874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176b5107-4763-404b-b51b-39bd741a6835-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2\" (UID: \"176b5107-4763-404b-b51b-39bd741a6835\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"
Sep 17 00:23:52.703963 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:52.703925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176b5107-4763-404b-b51b-39bd741a6835-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2\" (UID: \"176b5107-4763-404b-b51b-39bd741a6835\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"
Sep 17 00:23:52.704128 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:52.703963    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77pbk\" (UniqueName: \"kubernetes.io/projected/176b5107-4763-404b-b51b-39bd741a6835-kube-api-access-77pbk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2\" (UID: \"176b5107-4763-404b-b51b-39bd741a6835\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"
Sep 17 00:23:52.704170 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:52.704146    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176b5107-4763-404b-b51b-39bd741a6835-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2\" (UID: \"176b5107-4763-404b-b51b-39bd741a6835\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"
Sep 17 00:23:52.704330 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:52.704304    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176b5107-4763-404b-b51b-39bd741a6835-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2\" (UID: \"176b5107-4763-404b-b51b-39bd741a6835\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"
Sep 17 00:23:52.704468 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:52.704328    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176b5107-4763-404b-b51b-39bd741a6835-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2\" (UID: \"176b5107-4763-404b-b51b-39bd741a6835\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"
Sep 17 00:23:52.706124 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:52.706101    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176b5107-4763-404b-b51b-39bd741a6835-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2\" (UID: \"176b5107-4763-404b-b51b-39bd741a6835\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"
Sep 17 00:23:52.706542 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:52.706521    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176b5107-4763-404b-b51b-39bd741a6835-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2\" (UID: \"176b5107-4763-404b-b51b-39bd741a6835\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"
Sep 17 00:23:52.712595 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:52.712574    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77pbk\" (UniqueName: \"kubernetes.io/projected/176b5107-4763-404b-b51b-39bd741a6835-kube-api-access-77pbk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2\" (UID: \"176b5107-4763-404b-b51b-39bd741a6835\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"
Sep 17 00:23:52.812244 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:52.812160    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"
Sep 17 00:23:53.084483 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:53.084454    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"]
Sep 17 00:23:53.086444 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:23:53.086388    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod176b5107_4763_404b_b51b_39bd741a6835.slice/crio-ac2102586d7e44cf691049ff59b0e3bd562b51b5c0c05e98cea81942361f90a4 WatchSource:0}: Error finding container ac2102586d7e44cf691049ff59b0e3bd562b51b5c0c05e98cea81942361f90a4: Status 404 returned error can't find the container with id ac2102586d7e44cf691049ff59b0e3bd562b51b5c0c05e98cea81942361f90a4
Sep 17 00:23:53.277407 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:53.277377    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2" event={"ID":"176b5107-4763-404b-b51b-39bd741a6835","Type":"ContainerStarted","Data":"ac2102586d7e44cf691049ff59b0e3bd562b51b5c0c05e98cea81942361f90a4"}
Sep 17 00:23:54.284185 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:54.284146    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2" event={"ID":"176b5107-4763-404b-b51b-39bd741a6835","Type":"ContainerStarted","Data":"1bb08bfcd0cd54f1f3099fc43e8f718ce2f165014b6379b90df9023c82571272"}
Sep 17 00:23:54.300433 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:54.300370    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2" podStartSLOduration=2.132222967 podStartE2EDuration="2.300353705s" totalImagesPullingTime="168.130738ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:23:54.299589802 +0000 UTC m=+847.158082954" watchObservedRunningTime="2026-09-17 00:23:54.300353705 +0000 UTC m=+847.158846833"
Sep 17 00:23:54.703441 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:23:54.703387    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:24:02.813281 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:24:02.813242    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"
Sep 17 00:24:02.813949 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:24:02.813304    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"
Sep 17 00:24:02.826388 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:24:02.826356    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"
Sep 17 00:24:03.339689 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:24:03.339654    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2"
Sep 17 00:24:47.586866 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:24:47.586785    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/ovn-acl-logging/0.log"
Sep 17 00:24:47.588592 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:24:47.588568    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/ovn-acl-logging/0.log"
Sep 17 00:25:46.710538 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:46.710497    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:25:48.722950 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:48.722911    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj"]
Sep 17 00:25:48.733794 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:48.733762    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj"
Sep 17 00:25:48.736895 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:48.736867    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 17 00:25:48.737183 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:48.736867    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 17 00:25:48.737522 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:48.737095    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 00:25:48.737677 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:48.737654    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-2faea3-serving-cert\""
Sep 17 00:25:48.737777 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:48.737613    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2nbqn\""
Sep 17 00:25:48.738899 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:48.738872    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj"]
Sep 17 00:25:48.789566 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:48.789530    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljrgt\" (UniqueName: \"kubernetes.io/projected/95df6c2c-6af3-44c2-bed7-399abf2d29e9-kube-api-access-ljrgt\") pod \"maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj\" (UID: \"95df6c2c-6af3-44c2-bed7-399abf2d29e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj"
Sep 17 00:25:48.789744 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:48.789580    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95df6c2c-6af3-44c2-bed7-399abf2d29e9-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj\" (UID: \"95df6c2c-6af3-44c2-bed7-399abf2d29e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj"
Sep 17 00:25:48.789744 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:48.789609    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95df6c2c-6af3-44c2-bed7-399abf2d29e9-metrics-certs\") pod \"maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj\" (UID: \"95df6c2c-6af3-44c2-bed7-399abf2d29e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj"
Sep 17 00:25:48.789744 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:48.789681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95df6c2c-6af3-44c2-bed7-399abf2d29e9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj\" (UID: \"95df6c2c-6af3-44c2-bed7-399abf2d29e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj"
Sep 17 00:25:48.891095 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:48.891050    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljrgt\" (UniqueName: \"kubernetes.io/projected/95df6c2c-6af3-44c2-bed7-399abf2d29e9-kube-api-access-ljrgt\") pod \"maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj\" (UID: \"95df6c2c-6af3-44c2-bed7-399abf2d29e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj"
Sep 17 00:25:48.891274 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:48.891112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95df6c2c-6af3-44c2-bed7-399abf2d29e9-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj\" (UID: \"95df6c2c-6af3-44c2-bed7-399abf2d29e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj"
Sep 17 00:25:48.891274 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:48.891137    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95df6c2c-6af3-44c2-bed7-399abf2d29e9-metrics-certs\") pod \"maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj\" (UID: \"95df6c2c-6af3-44c2-bed7-399abf2d29e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj"
Sep 17 00:25:48.891274 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:48.891162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95df6c2c-6af3-44c2-bed7-399abf2d29e9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj\" (UID: \"95df6c2c-6af3-44c2-bed7-399abf2d29e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj"
Sep 17 00:25:48.891912 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:48.891892    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95df6c2c-6af3-44c2-bed7-399abf2d29e9-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj\" (UID: \"95df6c2c-6af3-44c2-bed7-399abf2d29e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj"
Sep 17 00:25:48.893613 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:48.893585    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95df6c2c-6af3-44c2-bed7-399abf2d29e9-metrics-certs\") pod \"maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj\" (UID: \"95df6c2c-6af3-44c2-bed7-399abf2d29e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj"
Sep 17 00:25:48.893721 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:48.893648    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95df6c2c-6af3-44c2-bed7-399abf2d29e9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj\" (UID: \"95df6c2c-6af3-44c2-bed7-399abf2d29e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj"
Sep 17 00:25:48.898552 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:48.898532    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljrgt\" (UniqueName: \"kubernetes.io/projected/95df6c2c-6af3-44c2-bed7-399abf2d29e9-kube-api-access-ljrgt\") pod \"maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj\" (UID: \"95df6c2c-6af3-44c2-bed7-399abf2d29e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj"
Sep 17 00:25:49.048469 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:49.048368    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj"
Sep 17 00:25:49.325234 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:49.325207    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj"]
Sep 17 00:25:49.327518 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:25:49.327480    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95df6c2c_6af3_44c2_bed7_399abf2d29e9.slice/crio-5a060d5a1ddc0d71d72a7dc46ba47e48a4d4d4526d7ca3809d51b42fbf336ba1 WatchSource:0}: Error finding container 5a060d5a1ddc0d71d72a7dc46ba47e48a4d4d4526d7ca3809d51b42fbf336ba1: Status 404 returned error can't find the container with id 5a060d5a1ddc0d71d72a7dc46ba47e48a4d4d4526d7ca3809d51b42fbf336ba1
Sep 17 00:25:49.881787 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:49.881753    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj" event={"ID":"95df6c2c-6af3-44c2-bed7-399abf2d29e9","Type":"ContainerStarted","Data":"5a060d5a1ddc0d71d72a7dc46ba47e48a4d4d4526d7ca3809d51b42fbf336ba1"}
Sep 17 00:25:50.679184 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:50.679147    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7840-60ff56"]
Sep 17 00:25:50.685968 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:50.685943    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7840-60ff56"
Sep 17 00:25:50.688498 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:50.688450    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7840-60ff56"]
Sep 17 00:25:50.688644 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:50.688552    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hf767\""
Sep 17 00:25:50.706934 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:50.706904    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppc9b\" (UniqueName: \"kubernetes.io/projected/ca535377-23de-4e92-883d-683044d04dbe-kube-api-access-ppc9b\") pod \"test-curl-7840-60ff56\" (UID: \"ca535377-23de-4e92-883d-683044d04dbe\") " pod="opendatahub/test-curl-7840-60ff56"
Sep 17 00:25:50.807626 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:50.807595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppc9b\" (UniqueName: \"kubernetes.io/projected/ca535377-23de-4e92-883d-683044d04dbe-kube-api-access-ppc9b\") pod \"test-curl-7840-60ff56\" (UID: \"ca535377-23de-4e92-883d-683044d04dbe\") " pod="opendatahub/test-curl-7840-60ff56"
Sep 17 00:25:50.815560 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:50.815526    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppc9b\" (UniqueName: \"kubernetes.io/projected/ca535377-23de-4e92-883d-683044d04dbe-kube-api-access-ppc9b\") pod \"test-curl-7840-60ff56\" (UID: \"ca535377-23de-4e92-883d-683044d04dbe\") " pod="opendatahub/test-curl-7840-60ff56"
Sep 17 00:25:51.000159 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:51.000123    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7840-60ff56"
Sep 17 00:25:51.895546 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:51.895511    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj" event={"ID":"95df6c2c-6af3-44c2-bed7-399abf2d29e9","Type":"ContainerStarted","Data":"b9a757ff59e15c83968c1ca41741b581d3258ed8e63844f9910b982abf543577"}
Sep 17 00:25:51.895716 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:51.895579    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj"
Sep 17 00:25:51.918027 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:51.917982    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj" podStartSLOduration=1.515795395 podStartE2EDuration="3.917966107s" totalImagesPullingTime="2.402170712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:25:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:25:51.915640306 +0000 UTC m=+964.774133464" watchObservedRunningTime="2026-09-17 00:25:51.917966107 +0000 UTC m=+964.776459242"
Sep 17 00:25:51.934889 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:51.934866    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7840-60ff56"]
Sep 17 00:25:51.936726 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:25:51.936696    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca535377_23de_4e92_883d_683044d04dbe.slice/crio-c1965229d3f0133e5f431ea84e591e416c01b568022da954ca9e6bd63d89e1fe WatchSource:0}: Error finding container c1965229d3f0133e5f431ea84e591e416c01b568022da954ca9e6bd63d89e1fe: Status 404 returned error can't find the container with id c1965229d3f0133e5f431ea84e591e416c01b568022da954ca9e6bd63d89e1fe
Sep 17 00:25:52.905961 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:52.905922    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7840-60ff56" event={"ID":"ca535377-23de-4e92-883d-683044d04dbe","Type":"ContainerStarted","Data":"c1965229d3f0133e5f431ea84e591e416c01b568022da954ca9e6bd63d89e1fe"}
Sep 17 00:25:54.918345 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:54.918309    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7840-60ff56" event={"ID":"ca535377-23de-4e92-883d-683044d04dbe","Type":"ContainerStarted","Data":"7c4ae71a83642e7dfd5da11493893199199ef0d60ac226c44d506cb3a7108535"}
Sep 17 00:25:54.932960 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:54.932915    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7840-60ff56" podStartSLOduration=2.636786521 podStartE2EDuration="4.932899661s" totalImagesPullingTime="2.29611314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:25:54.930916151 +0000 UTC m=+967.789409287" watchObservedRunningTime="2026-09-17 00:25:54.932899661 +0000 UTC m=+967.791392796"
Sep 17 00:25:55.616700 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:55.616619    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7840-60ff56"]
Sep 17 00:25:55.627154 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:55.627126    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7840-60ff56"]
Sep 17 00:25:55.923761 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:25:55.923670    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7840-60ff56" container="test-curl-7840-60ff56"
Sep 17 00:25:56.928873 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:56.928833    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7840-60ff56" podUID="ca535377-23de-4e92-883d-683044d04dbe" containerName="test-curl-7840-60ff56" containerID="cri-o://7c4ae71a83642e7dfd5da11493893199199ef0d60ac226c44d506cb3a7108535" gracePeriod=2
Sep 17 00:25:56.931049 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:25:56.931014    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7840-60ff56\" is forbidden: User \"system:node:ip-10-0-139-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-27.ec2.internal' and this object" podUID="ca535377-23de-4e92-883d-683044d04dbe" pod="opendatahub/test-curl-7840-60ff56"
Sep 17 00:25:57.606743 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:25:57.606700    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7840-60ff56\" is forbidden: User \"system:node:ip-10-0-139-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-27.ec2.internal' and this object" podUID="ca535377-23de-4e92-883d-683044d04dbe" pod="opendatahub/test-curl-7840-60ff56"
Sep 17 00:25:57.912121 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:57.912038    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj"
Sep 17 00:25:59.096475 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.096447    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7840-60ff56"
Sep 17 00:25:59.184722 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.184638    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppc9b\" (UniqueName: \"kubernetes.io/projected/ca535377-23de-4e92-883d-683044d04dbe-kube-api-access-ppc9b\") pod \"ca535377-23de-4e92-883d-683044d04dbe\" (UID: \"ca535377-23de-4e92-883d-683044d04dbe\") "
Sep 17 00:25:59.186919 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.186880    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca535377-23de-4e92-883d-683044d04dbe-kube-api-access-ppc9b" (OuterVolumeSpecName: "kube-api-access-ppc9b") pod "ca535377-23de-4e92-883d-683044d04dbe" (UID: "ca535377-23de-4e92-883d-683044d04dbe"). InnerVolumeSpecName "kube-api-access-ppc9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:25:59.217673 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.217645    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"]
Sep 17 00:25:59.217983 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.217971    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca535377-23de-4e92-883d-683044d04dbe" containerName="test-curl-7840-60ff56"
Sep 17 00:25:59.218026 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.217984    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca535377-23de-4e92-883d-683044d04dbe" containerName="test-curl-7840-60ff56"
Sep 17 00:25:59.218109 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.218051    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca535377-23de-4e92-883d-683044d04dbe" containerName="test-curl-7840-60ff56"
Sep 17 00:25:59.221112 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.221090    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"
Sep 17 00:25:59.223576 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.223554    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4df62ae2-serving-cert\""
Sep 17 00:25:59.232397 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.232374    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"]
Sep 17 00:25:59.286178 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.286146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac686a80-8954-4a13-81af-95a028315c49-metrics-certs\") pod \"maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9\" (UID: \"ac686a80-8954-4a13-81af-95a028315c49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"
Sep 17 00:25:59.286341 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.286207    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgb5t\" (UniqueName: \"kubernetes.io/projected/ac686a80-8954-4a13-81af-95a028315c49-kube-api-access-vgb5t\") pod \"maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9\" (UID: \"ac686a80-8954-4a13-81af-95a028315c49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"
Sep 17 00:25:59.286341 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.286251    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac686a80-8954-4a13-81af-95a028315c49-openshift-service-ca\") pod \"maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9\" (UID: \"ac686a80-8954-4a13-81af-95a028315c49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"
Sep 17 00:25:59.286341 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.286273    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac686a80-8954-4a13-81af-95a028315c49-maas-api-tls\") pod \"maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9\" (UID: \"ac686a80-8954-4a13-81af-95a028315c49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"
Sep 17 00:25:59.286341 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.286330    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppc9b\" (UniqueName: \"kubernetes.io/projected/ca535377-23de-4e92-883d-683044d04dbe-kube-api-access-ppc9b\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:25:59.387056 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.387018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac686a80-8954-4a13-81af-95a028315c49-metrics-certs\") pod \"maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9\" (UID: \"ac686a80-8954-4a13-81af-95a028315c49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"
Sep 17 00:25:59.387245 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.387079    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgb5t\" (UniqueName: \"kubernetes.io/projected/ac686a80-8954-4a13-81af-95a028315c49-kube-api-access-vgb5t\") pod \"maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9\" (UID: \"ac686a80-8954-4a13-81af-95a028315c49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"
Sep 17 00:25:59.387245 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.387104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac686a80-8954-4a13-81af-95a028315c49-openshift-service-ca\") pod \"maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9\" (UID: \"ac686a80-8954-4a13-81af-95a028315c49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"
Sep 17 00:25:59.387245 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.387126    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac686a80-8954-4a13-81af-95a028315c49-maas-api-tls\") pod \"maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9\" (UID: \"ac686a80-8954-4a13-81af-95a028315c49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"
Sep 17 00:25:59.387360 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:25:59.387254    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-serving-cert: secret "maas-api-e2e-ait-4df62ae2-serving-cert" not found
Sep 17 00:25:59.387360 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:25:59.387326    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac686a80-8954-4a13-81af-95a028315c49-maas-api-tls podName:ac686a80-8954-4a13-81af-95a028315c49 nodeName:}" failed. No retries permitted until 2026-09-17 00:25:59.887306827 +0000 UTC m=+972.745799945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac686a80-8954-4a13-81af-95a028315c49-maas-api-tls") pod "maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9" (UID: "ac686a80-8954-4a13-81af-95a028315c49") : secret "maas-api-e2e-ait-4df62ae2-serving-cert" not found
Sep 17 00:25:59.387772 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.387747    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac686a80-8954-4a13-81af-95a028315c49-openshift-service-ca\") pod \"maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9\" (UID: \"ac686a80-8954-4a13-81af-95a028315c49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"
Sep 17 00:25:59.389619 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.389596    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac686a80-8954-4a13-81af-95a028315c49-metrics-certs\") pod \"maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9\" (UID: \"ac686a80-8954-4a13-81af-95a028315c49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"
Sep 17 00:25:59.395315 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.395291    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgb5t\" (UniqueName: \"kubernetes.io/projected/ac686a80-8954-4a13-81af-95a028315c49-kube-api-access-vgb5t\") pod \"maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9\" (UID: \"ac686a80-8954-4a13-81af-95a028315c49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"
Sep 17 00:25:59.607798 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.607718    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca535377-23de-4e92-883d-683044d04dbe" path="/var/lib/kubelet/pods/ca535377-23de-4e92-883d-683044d04dbe/volumes"
Sep 17 00:25:59.891772 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.891677    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac686a80-8954-4a13-81af-95a028315c49-maas-api-tls\") pod \"maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9\" (UID: \"ac686a80-8954-4a13-81af-95a028315c49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"
Sep 17 00:25:59.894039 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.894019    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac686a80-8954-4a13-81af-95a028315c49-maas-api-tls\") pod \"maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9\" (UID: \"ac686a80-8954-4a13-81af-95a028315c49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"
Sep 17 00:25:59.952782 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.952747    2577 generic.go:356] "Generic (PLEG): container finished" podID="ca535377-23de-4e92-883d-683044d04dbe" containerID="7c4ae71a83642e7dfd5da11493893199199ef0d60ac226c44d506cb3a7108535" exitCode=137
Sep 17 00:25:59.952955 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.952792    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7840-60ff56"
Sep 17 00:25:59.952955 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.952821    2577 scope.go:117] "RemoveContainer" containerID="7c4ae71a83642e7dfd5da11493893199199ef0d60ac226c44d506cb3a7108535"
Sep 17 00:25:59.961349 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.961322    2577 scope.go:117] "RemoveContainer" containerID="7c4ae71a83642e7dfd5da11493893199199ef0d60ac226c44d506cb3a7108535"
Sep 17 00:25:59.961650 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:25:59.961623    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c4ae71a83642e7dfd5da11493893199199ef0d60ac226c44d506cb3a7108535\": container with ID starting with 7c4ae71a83642e7dfd5da11493893199199ef0d60ac226c44d506cb3a7108535 not found: ID does not exist" containerID="7c4ae71a83642e7dfd5da11493893199199ef0d60ac226c44d506cb3a7108535"
Sep 17 00:25:59.961747 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:25:59.961662    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c4ae71a83642e7dfd5da11493893199199ef0d60ac226c44d506cb3a7108535"} err="failed to get container status \"7c4ae71a83642e7dfd5da11493893199199ef0d60ac226c44d506cb3a7108535\": rpc error: code = NotFound desc = could not find container \"7c4ae71a83642e7dfd5da11493893199199ef0d60ac226c44d506cb3a7108535\": container with ID starting with 7c4ae71a83642e7dfd5da11493893199199ef0d60ac226c44d506cb3a7108535 not found: ID does not exist"
Sep 17 00:26:00.132173 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:00.132131    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"
Sep 17 00:26:00.407145 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:00.407120    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"]
Sep 17 00:26:00.408257 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:26:00.408230    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac686a80_8954_4a13_81af_95a028315c49.slice/crio-0a249fe933507dd2fd963c145af2572e08215dbb2e205bebda5ddd4f65a011c4 WatchSource:0}: Error finding container 0a249fe933507dd2fd963c145af2572e08215dbb2e205bebda5ddd4f65a011c4: Status 404 returned error can't find the container with id 0a249fe933507dd2fd963c145af2572e08215dbb2e205bebda5ddd4f65a011c4
Sep 17 00:26:00.958867 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:00.958826    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9" event={"ID":"ac686a80-8954-4a13-81af-95a028315c49","Type":"ContainerStarted","Data":"0a249fe933507dd2fd963c145af2572e08215dbb2e205bebda5ddd4f65a011c4"}
Sep 17 00:26:01.972570 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:01.972526    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9" event={"ID":"ac686a80-8954-4a13-81af-95a028315c49","Type":"ContainerStarted","Data":"040b8360c3454120c2e18f8a99c9d6f1e92eac96fb94fb644dfbdd92054d93da"}
Sep 17 00:26:01.972986 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:01.972686    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"
Sep 17 00:26:02.013330 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:02.013278    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9" podStartSLOduration=2.417263701 podStartE2EDuration="3.013261954s" totalImagesPullingTime="595.998253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:25:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:26:02.010715075 +0000 UTC m=+974.869208209" watchObservedRunningTime="2026-09-17 00:26:02.013261954 +0000 UTC m=+974.871755089"
Sep 17 00:26:03.798338 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:03.798304    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:26:04.018077 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:04.018029    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx"]
Sep 17 00:26:04.021918 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:04.021891    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx"
Sep 17 00:26:04.024159 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:04.024139    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-f492a5-serving-cert\""
Sep 17 00:26:04.032282 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:04.032257    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx"]
Sep 17 00:26:04.123503 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:04.123404    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2fef8e7-d128-4585-8910-983339e7ce25-openshift-service-ca\") pod \"maas-api-e2e-autores-f492a5-75977d9587-rbpwx\" (UID: \"b2fef8e7-d128-4585-8910-983339e7ce25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx"
Sep 17 00:26:04.123503 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:04.123459    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2fef8e7-d128-4585-8910-983339e7ce25-maas-api-tls\") pod \"maas-api-e2e-autores-f492a5-75977d9587-rbpwx\" (UID: \"b2fef8e7-d128-4585-8910-983339e7ce25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx"
Sep 17 00:26:04.123683 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:04.123549    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rjsw\" (UniqueName: \"kubernetes.io/projected/b2fef8e7-d128-4585-8910-983339e7ce25-kube-api-access-5rjsw\") pod \"maas-api-e2e-autores-f492a5-75977d9587-rbpwx\" (UID: \"b2fef8e7-d128-4585-8910-983339e7ce25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx"
Sep 17 00:26:04.123683 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:04.123585    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2fef8e7-d128-4585-8910-983339e7ce25-metrics-certs\") pod \"maas-api-e2e-autores-f492a5-75977d9587-rbpwx\" (UID: \"b2fef8e7-d128-4585-8910-983339e7ce25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx"
Sep 17 00:26:04.224218 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:04.224177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2fef8e7-d128-4585-8910-983339e7ce25-openshift-service-ca\") pod \"maas-api-e2e-autores-f492a5-75977d9587-rbpwx\" (UID: \"b2fef8e7-d128-4585-8910-983339e7ce25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx"
Sep 17 00:26:04.224382 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:04.224225    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2fef8e7-d128-4585-8910-983339e7ce25-maas-api-tls\") pod \"maas-api-e2e-autores-f492a5-75977d9587-rbpwx\" (UID: \"b2fef8e7-d128-4585-8910-983339e7ce25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx"
Sep 17 00:26:04.224382 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:04.224274    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rjsw\" (UniqueName: \"kubernetes.io/projected/b2fef8e7-d128-4585-8910-983339e7ce25-kube-api-access-5rjsw\") pod \"maas-api-e2e-autores-f492a5-75977d9587-rbpwx\" (UID: \"b2fef8e7-d128-4585-8910-983339e7ce25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx"
Sep 17 00:26:04.224382 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:04.224301    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2fef8e7-d128-4585-8910-983339e7ce25-metrics-certs\") pod \"maas-api-e2e-autores-f492a5-75977d9587-rbpwx\" (UID: \"b2fef8e7-d128-4585-8910-983339e7ce25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx"
Sep 17 00:26:04.224930 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:04.224907    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2fef8e7-d128-4585-8910-983339e7ce25-openshift-service-ca\") pod \"maas-api-e2e-autores-f492a5-75977d9587-rbpwx\" (UID: \"b2fef8e7-d128-4585-8910-983339e7ce25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx"
Sep 17 00:26:04.226765 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:04.226739    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2fef8e7-d128-4585-8910-983339e7ce25-metrics-certs\") pod \"maas-api-e2e-autores-f492a5-75977d9587-rbpwx\" (UID: \"b2fef8e7-d128-4585-8910-983339e7ce25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx"
Sep 17 00:26:04.226851 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:04.226789    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2fef8e7-d128-4585-8910-983339e7ce25-maas-api-tls\") pod \"maas-api-e2e-autores-f492a5-75977d9587-rbpwx\" (UID: \"b2fef8e7-d128-4585-8910-983339e7ce25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx"
Sep 17 00:26:04.231742 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:04.231721    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rjsw\" (UniqueName: \"kubernetes.io/projected/b2fef8e7-d128-4585-8910-983339e7ce25-kube-api-access-5rjsw\") pod \"maas-api-e2e-autores-f492a5-75977d9587-rbpwx\" (UID: \"b2fef8e7-d128-4585-8910-983339e7ce25\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx"
Sep 17 00:26:04.332590 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:04.332545    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx"
Sep 17 00:26:04.608372 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:04.608331    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx"]
Sep 17 00:26:04.610469 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:26:04.610441    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2fef8e7_d128_4585_8910_983339e7ce25.slice/crio-3f3f0c8c1654cbfd470877fe3586e043a4eea2a5d30a85f572a1ca83a428d14a WatchSource:0}: Error finding container 3f3f0c8c1654cbfd470877fe3586e043a4eea2a5d30a85f572a1ca83a428d14a: Status 404 returned error can't find the container with id 3f3f0c8c1654cbfd470877fe3586e043a4eea2a5d30a85f572a1ca83a428d14a
Sep 17 00:26:04.989078 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:04.989045    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx" event={"ID":"b2fef8e7-d128-4585-8910-983339e7ce25","Type":"ContainerStarted","Data":"3f3f0c8c1654cbfd470877fe3586e043a4eea2a5d30a85f572a1ca83a428d14a"}
Sep 17 00:26:05.996274 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:05.996232    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx" event={"ID":"b2fef8e7-d128-4585-8910-983339e7ce25","Type":"ContainerStarted","Data":"e5b38896b314f5667e3457585bd085516d05409d4d7f75d5923e78d91afa27de"}
Sep 17 00:26:05.996660 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:05.996304    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx"
Sep 17 00:26:06.013990 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:06.013942    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx" podStartSLOduration=2.197839654 podStartE2EDuration="3.013925897s" totalImagesPullingTime="816.086243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:26:06.012562284 +0000 UTC m=+978.871055420" watchObservedRunningTime="2026-09-17 00:26:06.013925897 +0000 UTC m=+978.872419031"
Sep 17 00:26:06.207599 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:06.207566    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb"]
Sep 17 00:26:06.210993 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:06.210975    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb"
Sep 17 00:26:06.213579 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:06.213557    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a146d6-serving-cert\""
Sep 17 00:26:06.220850 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:06.220820    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb"]
Sep 17 00:26:06.341093 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:06.340998    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/973c5c23-905c-4c92-bcdb-97b7600b449f-maas-api-tls\") pod \"maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb\" (UID: \"973c5c23-905c-4c92-bcdb-97b7600b449f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb"
Sep 17 00:26:06.341093 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:06.341066    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/973c5c23-905c-4c92-bcdb-97b7600b449f-metrics-certs\") pod \"maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb\" (UID: \"973c5c23-905c-4c92-bcdb-97b7600b449f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb"
Sep 17 00:26:06.341309 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:06.341119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4nn2\" (UniqueName: \"kubernetes.io/projected/973c5c23-905c-4c92-bcdb-97b7600b449f-kube-api-access-q4nn2\") pod \"maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb\" (UID: \"973c5c23-905c-4c92-bcdb-97b7600b449f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb"
Sep 17 00:26:06.341309 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:06.341179    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/973c5c23-905c-4c92-bcdb-97b7600b449f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb\" (UID: \"973c5c23-905c-4c92-bcdb-97b7600b449f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb"
Sep 17 00:26:06.441940 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:06.441906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/973c5c23-905c-4c92-bcdb-97b7600b449f-metrics-certs\") pod \"maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb\" (UID: \"973c5c23-905c-4c92-bcdb-97b7600b449f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb"
Sep 17 00:26:06.442144 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:06.441956    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4nn2\" (UniqueName: \"kubernetes.io/projected/973c5c23-905c-4c92-bcdb-97b7600b449f-kube-api-access-q4nn2\") pod \"maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb\" (UID: \"973c5c23-905c-4c92-bcdb-97b7600b449f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb"
Sep 17 00:26:06.442144 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:06.441985    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/973c5c23-905c-4c92-bcdb-97b7600b449f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb\" (UID: \"973c5c23-905c-4c92-bcdb-97b7600b449f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb"
Sep 17 00:26:06.442144 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:06.442012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/973c5c23-905c-4c92-bcdb-97b7600b449f-maas-api-tls\") pod \"maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb\" (UID: \"973c5c23-905c-4c92-bcdb-97b7600b449f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb"
Sep 17 00:26:06.442744 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:06.442722    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/973c5c23-905c-4c92-bcdb-97b7600b449f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb\" (UID: \"973c5c23-905c-4c92-bcdb-97b7600b449f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb"
Sep 17 00:26:06.444468 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:06.444446    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/973c5c23-905c-4c92-bcdb-97b7600b449f-metrics-certs\") pod \"maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb\" (UID: \"973c5c23-905c-4c92-bcdb-97b7600b449f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb"
Sep 17 00:26:06.444551 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:06.444495    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/973c5c23-905c-4c92-bcdb-97b7600b449f-maas-api-tls\") pod \"maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb\" (UID: \"973c5c23-905c-4c92-bcdb-97b7600b449f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb"
Sep 17 00:26:06.449131 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:06.449107    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4nn2\" (UniqueName: \"kubernetes.io/projected/973c5c23-905c-4c92-bcdb-97b7600b449f-kube-api-access-q4nn2\") pod \"maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb\" (UID: \"973c5c23-905c-4c92-bcdb-97b7600b449f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb"
Sep 17 00:26:06.523161 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:06.523116    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb"
Sep 17 00:26:06.797254 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:06.797228    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb"]
Sep 17 00:26:06.798215 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:26:06.798185    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod973c5c23_905c_4c92_bcdb_97b7600b449f.slice/crio-43a5cd982b9b6b584150331df87708f936072f8f058103f1098f3ef44d765a8f WatchSource:0}: Error finding container 43a5cd982b9b6b584150331df87708f936072f8f058103f1098f3ef44d765a8f: Status 404 returned error can't find the container with id 43a5cd982b9b6b584150331df87708f936072f8f058103f1098f3ef44d765a8f
Sep 17 00:26:07.003292 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:07.003254    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb" event={"ID":"973c5c23-905c-4c92-bcdb-97b7600b449f","Type":"ContainerStarted","Data":"43a5cd982b9b6b584150331df87708f936072f8f058103f1098f3ef44d765a8f"}
Sep 17 00:26:07.984006 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:07.983971    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"
Sep 17 00:26:08.011665 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:08.011619    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb" event={"ID":"973c5c23-905c-4c92-bcdb-97b7600b449f","Type":"ContainerStarted","Data":"8eaede97f49e2de9c220252c43dbde52ac460987c70dd2d4b320d016a7584e5b"}
Sep 17 00:26:08.012105 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:08.011728    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb"
Sep 17 00:26:08.030726 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:08.030663    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb" podStartSLOduration=1.648245385 podStartE2EDuration="2.030643285s" totalImagesPullingTime="382.3979ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:26:08.027789347 +0000 UTC m=+980.886282483" watchObservedRunningTime="2026-09-17 00:26:08.030643285 +0000 UTC m=+980.889136419"
Sep 17 00:26:12.008786 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:12.008714    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx"
Sep 17 00:26:14.022712 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:14.022683    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb"
Sep 17 00:26:24.399025 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:24.398984    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:26:36.283305 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.283269    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"]
Sep 17 00:26:36.289977 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.289955    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:26:36.292579 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.292550    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2faea3\"/\"kube-root-ca.crt\""
Sep 17 00:26:36.293230 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.293202    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"]
Sep 17 00:26:36.293694 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.293665    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2faea3\"/\"openshift-service-ca.crt\""
Sep 17 00:26:36.293811 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.293711    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2faea3\"/\"default-dockercfg-66bd9\""
Sep 17 00:26:36.293811 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.293746    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2faea3\"/\"simulator-w4-2faea3-kserve-self-signed-certs\""
Sep 17 00:26:36.410665 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.410624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e533b075-a947-45b0-bbbf-1ac95ade2554-home\") pod \"simulator-w4-2faea3-kserve-8f8d854f5-6skdg\" (UID: \"e533b075-a947-45b0-bbbf-1ac95ade2554\") " pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:26:36.410837 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.410730    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c4r8\" (UniqueName: \"kubernetes.io/projected/e533b075-a947-45b0-bbbf-1ac95ade2554-kube-api-access-4c4r8\") pod \"simulator-w4-2faea3-kserve-8f8d854f5-6skdg\" (UID: \"e533b075-a947-45b0-bbbf-1ac95ade2554\") " pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:26:36.410837 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.410771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e533b075-a947-45b0-bbbf-1ac95ade2554-tls-certs\") pod \"simulator-w4-2faea3-kserve-8f8d854f5-6skdg\" (UID: \"e533b075-a947-45b0-bbbf-1ac95ade2554\") " pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:26:36.410837 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.410826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e533b075-a947-45b0-bbbf-1ac95ade2554-dshm\") pod \"simulator-w4-2faea3-kserve-8f8d854f5-6skdg\" (UID: \"e533b075-a947-45b0-bbbf-1ac95ade2554\") " pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:26:36.410964 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.410862    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e533b075-a947-45b0-bbbf-1ac95ade2554-model-cache\") pod \"simulator-w4-2faea3-kserve-8f8d854f5-6skdg\" (UID: \"e533b075-a947-45b0-bbbf-1ac95ade2554\") " pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:26:36.410964 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.410879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e533b075-a947-45b0-bbbf-1ac95ade2554-tmp-dir\") pod \"simulator-w4-2faea3-kserve-8f8d854f5-6skdg\" (UID: \"e533b075-a947-45b0-bbbf-1ac95ade2554\") " pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:26:36.512097 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.512063    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e533b075-a947-45b0-bbbf-1ac95ade2554-tls-certs\") pod \"simulator-w4-2faea3-kserve-8f8d854f5-6skdg\" (UID: \"e533b075-a947-45b0-bbbf-1ac95ade2554\") " pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:26:36.512252 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.512119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e533b075-a947-45b0-bbbf-1ac95ade2554-dshm\") pod \"simulator-w4-2faea3-kserve-8f8d854f5-6skdg\" (UID: \"e533b075-a947-45b0-bbbf-1ac95ade2554\") " pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:26:36.512252 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.512150    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e533b075-a947-45b0-bbbf-1ac95ade2554-model-cache\") pod \"simulator-w4-2faea3-kserve-8f8d854f5-6skdg\" (UID: \"e533b075-a947-45b0-bbbf-1ac95ade2554\") " pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:26:36.512252 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.512165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e533b075-a947-45b0-bbbf-1ac95ade2554-tmp-dir\") pod \"simulator-w4-2faea3-kserve-8f8d854f5-6skdg\" (UID: \"e533b075-a947-45b0-bbbf-1ac95ade2554\") " pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:26:36.512252 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.512189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e533b075-a947-45b0-bbbf-1ac95ade2554-home\") pod \"simulator-w4-2faea3-kserve-8f8d854f5-6skdg\" (UID: \"e533b075-a947-45b0-bbbf-1ac95ade2554\") " pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:26:36.512505 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.512251    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c4r8\" (UniqueName: \"kubernetes.io/projected/e533b075-a947-45b0-bbbf-1ac95ade2554-kube-api-access-4c4r8\") pod \"simulator-w4-2faea3-kserve-8f8d854f5-6skdg\" (UID: \"e533b075-a947-45b0-bbbf-1ac95ade2554\") " pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:26:36.512677 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.512654    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e533b075-a947-45b0-bbbf-1ac95ade2554-home\") pod \"simulator-w4-2faea3-kserve-8f8d854f5-6skdg\" (UID: \"e533b075-a947-45b0-bbbf-1ac95ade2554\") " pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:26:36.512744 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.512677    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e533b075-a947-45b0-bbbf-1ac95ade2554-model-cache\") pod \"simulator-w4-2faea3-kserve-8f8d854f5-6skdg\" (UID: \"e533b075-a947-45b0-bbbf-1ac95ade2554\") " pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:26:36.512744 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.512727    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e533b075-a947-45b0-bbbf-1ac95ade2554-tmp-dir\") pod \"simulator-w4-2faea3-kserve-8f8d854f5-6skdg\" (UID: \"e533b075-a947-45b0-bbbf-1ac95ade2554\") " pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:26:36.514303 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.514278    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e533b075-a947-45b0-bbbf-1ac95ade2554-dshm\") pod \"simulator-w4-2faea3-kserve-8f8d854f5-6skdg\" (UID: \"e533b075-a947-45b0-bbbf-1ac95ade2554\") " pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:26:36.514627 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.514609    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e533b075-a947-45b0-bbbf-1ac95ade2554-tls-certs\") pod \"simulator-w4-2faea3-kserve-8f8d854f5-6skdg\" (UID: \"e533b075-a947-45b0-bbbf-1ac95ade2554\") " pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:26:36.519442 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.519406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c4r8\" (UniqueName: \"kubernetes.io/projected/e533b075-a947-45b0-bbbf-1ac95ade2554-kube-api-access-4c4r8\") pod \"simulator-w4-2faea3-kserve-8f8d854f5-6skdg\" (UID: \"e533b075-a947-45b0-bbbf-1ac95ade2554\") " pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:26:36.600731 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.600630    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:26:36.877985 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:36.877950    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"]
Sep 17 00:26:36.879886 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:26:36.879858    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode533b075_a947_45b0_bbbf_1ac95ade2554.slice/crio-7a8edd98336e4539604ea477114eef81e4d9994ba08d385e37ef21615c09a138 WatchSource:0}: Error finding container 7a8edd98336e4539604ea477114eef81e4d9994ba08d385e37ef21615c09a138: Status 404 returned error can't find the container with id 7a8edd98336e4539604ea477114eef81e4d9994ba08d385e37ef21615c09a138
Sep 17 00:26:37.172371 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:37.172271    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg" event={"ID":"e533b075-a947-45b0-bbbf-1ac95ade2554","Type":"ContainerStarted","Data":"ab7536cd8f3af8d7bfb7c13226d84cf3fd71979c255a4e23eae07e38f8de0cd6"}
Sep 17 00:26:37.172371 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:37.172313    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg" event={"ID":"e533b075-a947-45b0-bbbf-1ac95ade2554","Type":"ContainerStarted","Data":"7a8edd98336e4539604ea477114eef81e4d9994ba08d385e37ef21615c09a138"}
Sep 17 00:26:37.197327 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:37.197276    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg" podStartSLOduration=1.197260152 podStartE2EDuration="1.197260152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:26:37.194970201 +0000 UTC m=+1010.053463349" watchObservedRunningTime="2026-09-17 00:26:37.197260152 +0000 UTC m=+1010.055753337"
Sep 17 00:26:37.395980 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:37.395943    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"]
Sep 17 00:26:37.396587 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:37.396219    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9" podUID="ac686a80-8954-4a13-81af-95a028315c49" containerName="maas-api" containerID="cri-o://040b8360c3454120c2e18f8a99c9d6f1e92eac96fb94fb644dfbdd92054d93da" gracePeriod=30
Sep 17 00:26:37.640088 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:37.640065    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"
Sep 17 00:26:37.721557 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:37.721528    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac686a80-8954-4a13-81af-95a028315c49-maas-api-tls\") pod \"ac686a80-8954-4a13-81af-95a028315c49\" (UID: \"ac686a80-8954-4a13-81af-95a028315c49\") "
Sep 17 00:26:37.721557 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:37.721562    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac686a80-8954-4a13-81af-95a028315c49-metrics-certs\") pod \"ac686a80-8954-4a13-81af-95a028315c49\" (UID: \"ac686a80-8954-4a13-81af-95a028315c49\") "
Sep 17 00:26:37.721806 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:37.721617    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac686a80-8954-4a13-81af-95a028315c49-openshift-service-ca\") pod \"ac686a80-8954-4a13-81af-95a028315c49\" (UID: \"ac686a80-8954-4a13-81af-95a028315c49\") "
Sep 17 00:26:37.721806 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:37.721649    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgb5t\" (UniqueName: \"kubernetes.io/projected/ac686a80-8954-4a13-81af-95a028315c49-kube-api-access-vgb5t\") pod \"ac686a80-8954-4a13-81af-95a028315c49\" (UID: \"ac686a80-8954-4a13-81af-95a028315c49\") "
Sep 17 00:26:37.722164 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:37.722118    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac686a80-8954-4a13-81af-95a028315c49-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ac686a80-8954-4a13-81af-95a028315c49" (UID: "ac686a80-8954-4a13-81af-95a028315c49"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:26:37.724998 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:37.724968    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac686a80-8954-4a13-81af-95a028315c49-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ac686a80-8954-4a13-81af-95a028315c49" (UID: "ac686a80-8954-4a13-81af-95a028315c49"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:26:37.726698 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:37.726659    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac686a80-8954-4a13-81af-95a028315c49-kube-api-access-vgb5t" (OuterVolumeSpecName: "kube-api-access-vgb5t") pod "ac686a80-8954-4a13-81af-95a028315c49" (UID: "ac686a80-8954-4a13-81af-95a028315c49"). InnerVolumeSpecName "kube-api-access-vgb5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:26:37.726798 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:37.726721    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac686a80-8954-4a13-81af-95a028315c49-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ac686a80-8954-4a13-81af-95a028315c49" (UID: "ac686a80-8954-4a13-81af-95a028315c49"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:26:37.822782 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:37.822686    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac686a80-8954-4a13-81af-95a028315c49-maas-api-tls\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:26:37.822782 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:37.822724    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac686a80-8954-4a13-81af-95a028315c49-metrics-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:26:37.822782 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:37.822735    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac686a80-8954-4a13-81af-95a028315c49-openshift-service-ca\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:26:37.822782 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:37.822745    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgb5t\" (UniqueName: \"kubernetes.io/projected/ac686a80-8954-4a13-81af-95a028315c49-kube-api-access-vgb5t\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:26:38.179533 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:38.179440    2577 generic.go:356] "Generic (PLEG): container finished" podID="ac686a80-8954-4a13-81af-95a028315c49" containerID="040b8360c3454120c2e18f8a99c9d6f1e92eac96fb94fb644dfbdd92054d93da" exitCode=0
Sep 17 00:26:38.179533 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:38.179489    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"
Sep 17 00:26:38.179717 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:38.179520    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9" event={"ID":"ac686a80-8954-4a13-81af-95a028315c49","Type":"ContainerDied","Data":"040b8360c3454120c2e18f8a99c9d6f1e92eac96fb94fb644dfbdd92054d93da"}
Sep 17 00:26:38.179717 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:38.179589    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9" event={"ID":"ac686a80-8954-4a13-81af-95a028315c49","Type":"ContainerDied","Data":"0a249fe933507dd2fd963c145af2572e08215dbb2e205bebda5ddd4f65a011c4"}
Sep 17 00:26:38.179717 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:38.179625    2577 scope.go:117] "RemoveContainer" containerID="040b8360c3454120c2e18f8a99c9d6f1e92eac96fb94fb644dfbdd92054d93da"
Sep 17 00:26:38.189304 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:38.189280    2577 scope.go:117] "RemoveContainer" containerID="040b8360c3454120c2e18f8a99c9d6f1e92eac96fb94fb644dfbdd92054d93da"
Sep 17 00:26:38.189625 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:26:38.189605    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"040b8360c3454120c2e18f8a99c9d6f1e92eac96fb94fb644dfbdd92054d93da\": container with ID starting with 040b8360c3454120c2e18f8a99c9d6f1e92eac96fb94fb644dfbdd92054d93da not found: ID does not exist" containerID="040b8360c3454120c2e18f8a99c9d6f1e92eac96fb94fb644dfbdd92054d93da"
Sep 17 00:26:38.189704 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:38.189633    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040b8360c3454120c2e18f8a99c9d6f1e92eac96fb94fb644dfbdd92054d93da"} err="failed to get container status \"040b8360c3454120c2e18f8a99c9d6f1e92eac96fb94fb644dfbdd92054d93da\": rpc error: code = NotFound desc = could not find container \"040b8360c3454120c2e18f8a99c9d6f1e92eac96fb94fb644dfbdd92054d93da\": container with ID starting with 040b8360c3454120c2e18f8a99c9d6f1e92eac96fb94fb644dfbdd92054d93da not found: ID does not exist"
Sep 17 00:26:38.205709 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:38.205674    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"]
Sep 17 00:26:38.214394 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:38.214358    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4df62ae2-86d46bfb49-9mrm9"]
Sep 17 00:26:39.607724 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:39.607695    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac686a80-8954-4a13-81af-95a028315c49" path="/var/lib/kubelet/pods/ac686a80-8954-4a13-81af-95a028315c49/volumes"
Sep 17 00:26:42.750825 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:42.750785    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb"]
Sep 17 00:26:42.751327 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:42.751310    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac686a80-8954-4a13-81af-95a028315c49" containerName="maas-api"
Sep 17 00:26:42.751438 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:42.751329    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac686a80-8954-4a13-81af-95a028315c49" containerName="maas-api"
Sep 17 00:26:42.751438 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:42.751406    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac686a80-8954-4a13-81af-95a028315c49" containerName="maas-api"
Sep 17 00:26:42.754194 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:42.754174    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb"
Sep 17 00:26:42.754791 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:42.754762    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fe6a837-88f8-4a5c-8427-35c866fe2383-maas-api-tls\") pod \"maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb\" (UID: \"2fe6a837-88f8-4a5c-8427-35c866fe2383\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb"
Sep 17 00:26:42.754912 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:42.754811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe6a837-88f8-4a5c-8427-35c866fe2383-openshift-service-ca\") pod \"maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb\" (UID: \"2fe6a837-88f8-4a5c-8427-35c866fe2383\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb"
Sep 17 00:26:42.754912 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:42.754893    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxw7r\" (UniqueName: \"kubernetes.io/projected/2fe6a837-88f8-4a5c-8427-35c866fe2383-kube-api-access-fxw7r\") pod \"maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb\" (UID: \"2fe6a837-88f8-4a5c-8427-35c866fe2383\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb"
Sep 17 00:26:42.755024 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:42.754962    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fe6a837-88f8-4a5c-8427-35c866fe2383-metrics-certs\") pod \"maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb\" (UID: \"2fe6a837-88f8-4a5c-8427-35c866fe2383\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb"
Sep 17 00:26:42.756697 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:42.756673    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-3932eef5-serving-cert\""
Sep 17 00:26:42.766566 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:42.766504    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb"]
Sep 17 00:26:42.855457 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:42.855405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxw7r\" (UniqueName: \"kubernetes.io/projected/2fe6a837-88f8-4a5c-8427-35c866fe2383-kube-api-access-fxw7r\") pod \"maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb\" (UID: \"2fe6a837-88f8-4a5c-8427-35c866fe2383\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb"
Sep 17 00:26:42.855636 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:42.855499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fe6a837-88f8-4a5c-8427-35c866fe2383-metrics-certs\") pod \"maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb\" (UID: \"2fe6a837-88f8-4a5c-8427-35c866fe2383\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb"
Sep 17 00:26:42.855636 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:42.855538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fe6a837-88f8-4a5c-8427-35c866fe2383-maas-api-tls\") pod \"maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb\" (UID: \"2fe6a837-88f8-4a5c-8427-35c866fe2383\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb"
Sep 17 00:26:42.855636 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:42.855562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe6a837-88f8-4a5c-8427-35c866fe2383-openshift-service-ca\") pod \"maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb\" (UID: \"2fe6a837-88f8-4a5c-8427-35c866fe2383\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb"
Sep 17 00:26:42.856213 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:42.856186    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe6a837-88f8-4a5c-8427-35c866fe2383-openshift-service-ca\") pod \"maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb\" (UID: \"2fe6a837-88f8-4a5c-8427-35c866fe2383\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb"
Sep 17 00:26:42.858010 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:42.857972    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fe6a837-88f8-4a5c-8427-35c866fe2383-maas-api-tls\") pod \"maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb\" (UID: \"2fe6a837-88f8-4a5c-8427-35c866fe2383\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb"
Sep 17 00:26:42.858131 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:42.858107    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fe6a837-88f8-4a5c-8427-35c866fe2383-metrics-certs\") pod \"maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb\" (UID: \"2fe6a837-88f8-4a5c-8427-35c866fe2383\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb"
Sep 17 00:26:42.863179 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:42.863159    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxw7r\" (UniqueName: \"kubernetes.io/projected/2fe6a837-88f8-4a5c-8427-35c866fe2383-kube-api-access-fxw7r\") pod \"maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb\" (UID: \"2fe6a837-88f8-4a5c-8427-35c866fe2383\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb"
Sep 17 00:26:43.018720 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:43.018627    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:26:43.067702 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:43.067667    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb"
Sep 17 00:26:43.342670 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:43.342644    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb"]
Sep 17 00:26:43.344235 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:26:43.344204    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe6a837_88f8_4a5c_8427_35c866fe2383.slice/crio-b840bccee9ee561ce7d5673defcd6646149dd86adcfc51a499dce51275bc69a6 WatchSource:0}: Error finding container b840bccee9ee561ce7d5673defcd6646149dd86adcfc51a499dce51275bc69a6: Status 404 returned error can't find the container with id b840bccee9ee561ce7d5673defcd6646149dd86adcfc51a499dce51275bc69a6
Sep 17 00:26:43.346082 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:43.346059    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 00:26:44.217742 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:44.217701    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb" event={"ID":"2fe6a837-88f8-4a5c-8427-35c866fe2383","Type":"ContainerStarted","Data":"7a7eef94cbd6ad7261e8cdb0a7cf6232cc0aba3c9c937924f9078ad1fbcd5b07"}
Sep 17 00:26:44.218106 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:44.217763    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb" event={"ID":"2fe6a837-88f8-4a5c-8427-35c866fe2383","Type":"ContainerStarted","Data":"b840bccee9ee561ce7d5673defcd6646149dd86adcfc51a499dce51275bc69a6"}
Sep 17 00:26:44.218106 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:44.217794    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb"
Sep 17 00:26:44.237013 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:44.236926    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb" podStartSLOduration=1.612450428 podStartE2EDuration="2.236909982s" totalImagesPullingTime="624.459554ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:26:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:26:44.234459621 +0000 UTC m=+1017.092952756" watchObservedRunningTime="2026-09-17 00:26:44.236909982 +0000 UTC m=+1017.095403116"
Sep 17 00:26:46.601240 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:46.601187    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:26:46.601750 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:46.601252    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:26:46.613735 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:46.613708    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:26:47.243852 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:47.243823    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:26:49.783030 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.782995    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"]
Sep 17 00:26:49.786219 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.786202    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:26:49.788603 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.788581    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-cfe049\"/\"kube-root-ca.crt\""
Sep 17 00:26:49.789735 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.789719    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-cfe049\"/\"openshift-service-ca.crt\""
Sep 17 00:26:49.789833 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.789742    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-cfe049\"/\"default-dockercfg-zhhbp\""
Sep 17 00:26:49.789900 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.789864    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-cfe049\"/\"simulator-w0-cfe049-kserve-self-signed-certs\""
Sep 17 00:26:49.796838 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.796819    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"]
Sep 17 00:26:49.822991 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.822963    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqcxf\" (UniqueName: \"kubernetes.io/projected/97a1c787-acbc-4d1c-9657-b79b9db6de85-kube-api-access-nqcxf\") pod \"simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8\" (UID: \"97a1c787-acbc-4d1c-9657-b79b9db6de85\") " pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:26:49.823136 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.823019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a1c787-acbc-4d1c-9657-b79b9db6de85-dshm\") pod \"simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8\" (UID: \"97a1c787-acbc-4d1c-9657-b79b9db6de85\") " pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:26:49.823136 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.823065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a1c787-acbc-4d1c-9657-b79b9db6de85-model-cache\") pod \"simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8\" (UID: \"97a1c787-acbc-4d1c-9657-b79b9db6de85\") " pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:26:49.823136 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.823118    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a1c787-acbc-4d1c-9657-b79b9db6de85-tmp-dir\") pod \"simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8\" (UID: \"97a1c787-acbc-4d1c-9657-b79b9db6de85\") " pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:26:49.823289 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.823142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a1c787-acbc-4d1c-9657-b79b9db6de85-tls-certs\") pod \"simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8\" (UID: \"97a1c787-acbc-4d1c-9657-b79b9db6de85\") " pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:26:49.823289 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.823170    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a1c787-acbc-4d1c-9657-b79b9db6de85-home\") pod \"simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8\" (UID: \"97a1c787-acbc-4d1c-9657-b79b9db6de85\") " pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:26:49.923920 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.923882    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqcxf\" (UniqueName: \"kubernetes.io/projected/97a1c787-acbc-4d1c-9657-b79b9db6de85-kube-api-access-nqcxf\") pod \"simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8\" (UID: \"97a1c787-acbc-4d1c-9657-b79b9db6de85\") " pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:26:49.924111 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.923930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a1c787-acbc-4d1c-9657-b79b9db6de85-dshm\") pod \"simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8\" (UID: \"97a1c787-acbc-4d1c-9657-b79b9db6de85\") " pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:26:49.924111 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.923952    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a1c787-acbc-4d1c-9657-b79b9db6de85-model-cache\") pod \"simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8\" (UID: \"97a1c787-acbc-4d1c-9657-b79b9db6de85\") " pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:26:49.924111 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.924000    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a1c787-acbc-4d1c-9657-b79b9db6de85-tmp-dir\") pod \"simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8\" (UID: \"97a1c787-acbc-4d1c-9657-b79b9db6de85\") " pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:26:49.924111 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.924036    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a1c787-acbc-4d1c-9657-b79b9db6de85-tls-certs\") pod \"simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8\" (UID: \"97a1c787-acbc-4d1c-9657-b79b9db6de85\") " pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:26:49.924111 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.924074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a1c787-acbc-4d1c-9657-b79b9db6de85-home\") pod \"simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8\" (UID: \"97a1c787-acbc-4d1c-9657-b79b9db6de85\") " pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:26:49.924500 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.924462    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a1c787-acbc-4d1c-9657-b79b9db6de85-tmp-dir\") pod \"simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8\" (UID: \"97a1c787-acbc-4d1c-9657-b79b9db6de85\") " pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:26:49.924500 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.924495    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a1c787-acbc-4d1c-9657-b79b9db6de85-model-cache\") pod \"simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8\" (UID: \"97a1c787-acbc-4d1c-9657-b79b9db6de85\") " pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:26:49.924659 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.924505    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a1c787-acbc-4d1c-9657-b79b9db6de85-home\") pod \"simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8\" (UID: \"97a1c787-acbc-4d1c-9657-b79b9db6de85\") " pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:26:49.926238 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.926209    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a1c787-acbc-4d1c-9657-b79b9db6de85-dshm\") pod \"simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8\" (UID: \"97a1c787-acbc-4d1c-9657-b79b9db6de85\") " pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:26:49.926506 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.926488    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a1c787-acbc-4d1c-9657-b79b9db6de85-tls-certs\") pod \"simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8\" (UID: \"97a1c787-acbc-4d1c-9657-b79b9db6de85\") " pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:26:49.931106 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:49.931080    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqcxf\" (UniqueName: \"kubernetes.io/projected/97a1c787-acbc-4d1c-9657-b79b9db6de85-kube-api-access-nqcxf\") pod \"simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8\" (UID: \"97a1c787-acbc-4d1c-9657-b79b9db6de85\") " pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:26:50.096299 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:50.096210    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:26:50.228486 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:50.228452    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb"
Sep 17 00:26:50.367938 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:50.367845    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"]
Sep 17 00:26:50.371397 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:26:50.371371    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97a1c787_acbc_4d1c_9657_b79b9db6de85.slice/crio-c2c6929a348faeee086af8ed7393e61906864ee0ee2ed1abe10029199eab07dc WatchSource:0}: Error finding container c2c6929a348faeee086af8ed7393e61906864ee0ee2ed1abe10029199eab07dc: Status 404 returned error can't find the container with id c2c6929a348faeee086af8ed7393e61906864ee0ee2ed1abe10029199eab07dc
Sep 17 00:26:51.265901 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:51.265855    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8" event={"ID":"97a1c787-acbc-4d1c-9657-b79b9db6de85","Type":"ContainerStarted","Data":"7baf95da4bc3e9f9df760a9b8d92891f81a2eb5aff8267189512ba7ac7720bf0"}
Sep 17 00:26:51.265901 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:51.265909    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8" event={"ID":"97a1c787-acbc-4d1c-9657-b79b9db6de85","Type":"ContainerStarted","Data":"c2c6929a348faeee086af8ed7393e61906864ee0ee2ed1abe10029199eab07dc"}
Sep 17 00:26:51.285024 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:26:51.284969    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8" podStartSLOduration=2.284952465 podStartE2EDuration="2.284952465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:26:51.282605645 +0000 UTC m=+1024.141098805" watchObservedRunningTime="2026-09-17 00:26:51.284952465 +0000 UTC m=+1024.143445599"
Sep 17 00:27:00.096814 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:00.096761    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:27:00.097297 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:00.096948    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:27:00.109371 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:00.109342    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:27:00.325657 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:00.325622    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:27:05.304276 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:05.304236    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:27:12.555068 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:12.555027    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb"]
Sep 17 00:27:12.555666 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:12.555305    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb" podUID="2fe6a837-88f8-4a5c-8427-35c866fe2383" containerName="maas-api" containerID="cri-o://7a7eef94cbd6ad7261e8cdb0a7cf6232cc0aba3c9c937924f9078ad1fbcd5b07" gracePeriod=30
Sep 17 00:27:12.805976 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:12.805910    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb"
Sep 17 00:27:12.938062 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:12.938024    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fe6a837-88f8-4a5c-8427-35c866fe2383-maas-api-tls\") pod \"2fe6a837-88f8-4a5c-8427-35c866fe2383\" (UID: \"2fe6a837-88f8-4a5c-8427-35c866fe2383\") "
Sep 17 00:27:12.938256 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:12.938088    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fe6a837-88f8-4a5c-8427-35c866fe2383-metrics-certs\") pod \"2fe6a837-88f8-4a5c-8427-35c866fe2383\" (UID: \"2fe6a837-88f8-4a5c-8427-35c866fe2383\") "
Sep 17 00:27:12.938256 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:12.938158    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxw7r\" (UniqueName: \"kubernetes.io/projected/2fe6a837-88f8-4a5c-8427-35c866fe2383-kube-api-access-fxw7r\") pod \"2fe6a837-88f8-4a5c-8427-35c866fe2383\" (UID: \"2fe6a837-88f8-4a5c-8427-35c866fe2383\") "
Sep 17 00:27:12.938256 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:12.938184    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe6a837-88f8-4a5c-8427-35c866fe2383-openshift-service-ca\") pod \"2fe6a837-88f8-4a5c-8427-35c866fe2383\" (UID: \"2fe6a837-88f8-4a5c-8427-35c866fe2383\") "
Sep 17 00:27:12.938638 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:12.938604    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fe6a837-88f8-4a5c-8427-35c866fe2383-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2fe6a837-88f8-4a5c-8427-35c866fe2383" (UID: "2fe6a837-88f8-4a5c-8427-35c866fe2383"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:27:12.940287 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:12.940255    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fe6a837-88f8-4a5c-8427-35c866fe2383-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2fe6a837-88f8-4a5c-8427-35c866fe2383" (UID: "2fe6a837-88f8-4a5c-8427-35c866fe2383"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:27:12.940386 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:12.940325    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe6a837-88f8-4a5c-8427-35c866fe2383-kube-api-access-fxw7r" (OuterVolumeSpecName: "kube-api-access-fxw7r") pod "2fe6a837-88f8-4a5c-8427-35c866fe2383" (UID: "2fe6a837-88f8-4a5c-8427-35c866fe2383"). InnerVolumeSpecName "kube-api-access-fxw7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:27:12.940386 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:12.940332    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fe6a837-88f8-4a5c-8427-35c866fe2383-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2fe6a837-88f8-4a5c-8427-35c866fe2383" (UID: "2fe6a837-88f8-4a5c-8427-35c866fe2383"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:27:13.039329 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:13.039291    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxw7r\" (UniqueName: \"kubernetes.io/projected/2fe6a837-88f8-4a5c-8427-35c866fe2383-kube-api-access-fxw7r\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:27:13.039329 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:13.039324    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe6a837-88f8-4a5c-8427-35c866fe2383-openshift-service-ca\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:27:13.039329 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:13.039334    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fe6a837-88f8-4a5c-8427-35c866fe2383-maas-api-tls\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:27:13.039630 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:13.039343    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fe6a837-88f8-4a5c-8427-35c866fe2383-metrics-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:27:13.390181 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:13.390135    2577 generic.go:356] "Generic (PLEG): container finished" podID="2fe6a837-88f8-4a5c-8427-35c866fe2383" containerID="7a7eef94cbd6ad7261e8cdb0a7cf6232cc0aba3c9c937924f9078ad1fbcd5b07" exitCode=0
Sep 17 00:27:13.390181 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:13.390185    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb"
Sep 17 00:27:13.390439 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:13.390223    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb" event={"ID":"2fe6a837-88f8-4a5c-8427-35c866fe2383","Type":"ContainerDied","Data":"7a7eef94cbd6ad7261e8cdb0a7cf6232cc0aba3c9c937924f9078ad1fbcd5b07"}
Sep 17 00:27:13.390439 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:13.390279    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb" event={"ID":"2fe6a837-88f8-4a5c-8427-35c866fe2383","Type":"ContainerDied","Data":"b840bccee9ee561ce7d5673defcd6646149dd86adcfc51a499dce51275bc69a6"}
Sep 17 00:27:13.390439 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:13.390301    2577 scope.go:117] "RemoveContainer" containerID="7a7eef94cbd6ad7261e8cdb0a7cf6232cc0aba3c9c937924f9078ad1fbcd5b07"
Sep 17 00:27:13.400282 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:13.400255    2577 scope.go:117] "RemoveContainer" containerID="7a7eef94cbd6ad7261e8cdb0a7cf6232cc0aba3c9c937924f9078ad1fbcd5b07"
Sep 17 00:27:13.400624 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:27:13.400601    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a7eef94cbd6ad7261e8cdb0a7cf6232cc0aba3c9c937924f9078ad1fbcd5b07\": container with ID starting with 7a7eef94cbd6ad7261e8cdb0a7cf6232cc0aba3c9c937924f9078ad1fbcd5b07 not found: ID does not exist" containerID="7a7eef94cbd6ad7261e8cdb0a7cf6232cc0aba3c9c937924f9078ad1fbcd5b07"
Sep 17 00:27:13.400700 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:13.400633    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a7eef94cbd6ad7261e8cdb0a7cf6232cc0aba3c9c937924f9078ad1fbcd5b07"} err="failed to get container status \"7a7eef94cbd6ad7261e8cdb0a7cf6232cc0aba3c9c937924f9078ad1fbcd5b07\": rpc error: code = NotFound desc = could not find container \"7a7eef94cbd6ad7261e8cdb0a7cf6232cc0aba3c9c937924f9078ad1fbcd5b07\": container with ID starting with 7a7eef94cbd6ad7261e8cdb0a7cf6232cc0aba3c9c937924f9078ad1fbcd5b07 not found: ID does not exist"
Sep 17 00:27:13.411885 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:13.411852    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb"]
Sep 17 00:27:13.415495 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:13.415462    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3932eef5-5d68b76f94-v65jb"]
Sep 17 00:27:13.610812 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:13.610774    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fe6a837-88f8-4a5c-8427-35c866fe2383" path="/var/lib/kubelet/pods/2fe6a837-88f8-4a5c-8427-35c866fe2383/volumes"
Sep 17 00:27:17.511443 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:17.511392    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6"]
Sep 17 00:27:17.511991 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:17.511970    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fe6a837-88f8-4a5c-8427-35c866fe2383" containerName="maas-api"
Sep 17 00:27:17.512073 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:17.511995    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fe6a837-88f8-4a5c-8427-35c866fe2383" containerName="maas-api"
Sep 17 00:27:17.512135 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:17.512100    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fe6a837-88f8-4a5c-8427-35c866fe2383" containerName="maas-api"
Sep 17 00:27:17.514118 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:17.514096    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6"
Sep 17 00:27:17.516709 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:17.516685    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-56ff2313-serving-cert\""
Sep 17 00:27:17.524715 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:17.524686    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6"]
Sep 17 00:27:17.683936 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:17.683900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2400633-7a6e-4ca0-8771-77abb2192325-maas-api-tls\") pod \"maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6\" (UID: \"d2400633-7a6e-4ca0-8771-77abb2192325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6"
Sep 17 00:27:17.683936 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:17.683939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2400633-7a6e-4ca0-8771-77abb2192325-metrics-certs\") pod \"maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6\" (UID: \"d2400633-7a6e-4ca0-8771-77abb2192325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6"
Sep 17 00:27:17.684183 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:17.683981    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwr4t\" (UniqueName: \"kubernetes.io/projected/d2400633-7a6e-4ca0-8771-77abb2192325-kube-api-access-fwr4t\") pod \"maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6\" (UID: \"d2400633-7a6e-4ca0-8771-77abb2192325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6"
Sep 17 00:27:17.684183 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:17.684059    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2400633-7a6e-4ca0-8771-77abb2192325-openshift-service-ca\") pod \"maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6\" (UID: \"d2400633-7a6e-4ca0-8771-77abb2192325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6"
Sep 17 00:27:17.785472 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:17.785354    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwr4t\" (UniqueName: \"kubernetes.io/projected/d2400633-7a6e-4ca0-8771-77abb2192325-kube-api-access-fwr4t\") pod \"maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6\" (UID: \"d2400633-7a6e-4ca0-8771-77abb2192325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6"
Sep 17 00:27:17.785649 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:17.785484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2400633-7a6e-4ca0-8771-77abb2192325-openshift-service-ca\") pod \"maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6\" (UID: \"d2400633-7a6e-4ca0-8771-77abb2192325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6"
Sep 17 00:27:17.785649 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:17.785574    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2400633-7a6e-4ca0-8771-77abb2192325-maas-api-tls\") pod \"maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6\" (UID: \"d2400633-7a6e-4ca0-8771-77abb2192325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6"
Sep 17 00:27:17.785649 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:17.785619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2400633-7a6e-4ca0-8771-77abb2192325-metrics-certs\") pod \"maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6\" (UID: \"d2400633-7a6e-4ca0-8771-77abb2192325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6"
Sep 17 00:27:17.786233 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:17.786203    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2400633-7a6e-4ca0-8771-77abb2192325-openshift-service-ca\") pod \"maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6\" (UID: \"d2400633-7a6e-4ca0-8771-77abb2192325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6"
Sep 17 00:27:17.788392 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:17.788362    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2400633-7a6e-4ca0-8771-77abb2192325-metrics-certs\") pod \"maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6\" (UID: \"d2400633-7a6e-4ca0-8771-77abb2192325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6"
Sep 17 00:27:17.788522 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:17.788438    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2400633-7a6e-4ca0-8771-77abb2192325-maas-api-tls\") pod \"maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6\" (UID: \"d2400633-7a6e-4ca0-8771-77abb2192325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6"
Sep 17 00:27:17.793144 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:17.793120    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwr4t\" (UniqueName: \"kubernetes.io/projected/d2400633-7a6e-4ca0-8771-77abb2192325-kube-api-access-fwr4t\") pod \"maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6\" (UID: \"d2400633-7a6e-4ca0-8771-77abb2192325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6"
Sep 17 00:27:17.826977 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:17.826947    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6"
Sep 17 00:27:18.120484 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:18.120453    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6"]
Sep 17 00:27:18.122184 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:27:18.122157    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2400633_7a6e_4ca0_8771_77abb2192325.slice/crio-b77b73cfa37a2399d1aa48451c19384f589a80edb25d8cd1142e76ffa8afba1e WatchSource:0}: Error finding container b77b73cfa37a2399d1aa48451c19384f589a80edb25d8cd1142e76ffa8afba1e: Status 404 returned error can't find the container with id b77b73cfa37a2399d1aa48451c19384f589a80edb25d8cd1142e76ffa8afba1e
Sep 17 00:27:18.427202 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:18.427107    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6" event={"ID":"d2400633-7a6e-4ca0-8771-77abb2192325","Type":"ContainerStarted","Data":"b77b73cfa37a2399d1aa48451c19384f589a80edb25d8cd1142e76ffa8afba1e"}
Sep 17 00:27:19.095572 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:19.095536    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:27:19.433624 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:19.433586    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6" event={"ID":"d2400633-7a6e-4ca0-8771-77abb2192325","Type":"ContainerStarted","Data":"281a9c1ba8907b3c2bf4498883030938f345ca8c683a07a3cd29c350780f79bf"}
Sep 17 00:27:19.433804 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:19.433659    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6"
Sep 17 00:27:19.453571 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:19.453508    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6" podStartSLOduration=1.9566951700000002 podStartE2EDuration="2.453483191s" totalImagesPullingTime="496.788021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:27:19.450006725 +0000 UTC m=+1052.308499862" watchObservedRunningTime="2026-09-17 00:27:19.453483191 +0000 UTC m=+1052.311976327"
Sep 17 00:27:25.444816 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:25.444783    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6"
Sep 17 00:27:36.395530 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:36.395490    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:27:43.799345 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:43.799151    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"]
Sep 17 00:27:43.812009 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:43.811972    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:27:43.817609 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:43.817578    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-76944c\"/\"test-model-76944c-kserve-self-signed-certs\""
Sep 17 00:27:43.818302 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:43.818107    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-76944c\"/\"openshift-service-ca.crt\""
Sep 17 00:27:43.818465 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:43.818339    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-76944c\"/\"kube-root-ca.crt\""
Sep 17 00:27:43.818545 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:43.818459    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-76944c\"/\"default-dockercfg-zhldm\""
Sep 17 00:27:43.819637 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:43.819612    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"]
Sep 17 00:27:43.916938 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:43.916905    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbkm8\" (UniqueName: \"kubernetes.io/projected/850c3459-5c5f-4b41-9aeb-905c098d4189-kube-api-access-xbkm8\") pod \"test-model-76944c-kserve-6bf9bf4b67-6jgxz\" (UID: \"850c3459-5c5f-4b41-9aeb-905c098d4189\") " pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:27:43.916938 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:43.916942    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/850c3459-5c5f-4b41-9aeb-905c098d4189-home\") pod \"test-model-76944c-kserve-6bf9bf4b67-6jgxz\" (UID: \"850c3459-5c5f-4b41-9aeb-905c098d4189\") " pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:27:43.917139 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:43.917008    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/850c3459-5c5f-4b41-9aeb-905c098d4189-tls-certs\") pod \"test-model-76944c-kserve-6bf9bf4b67-6jgxz\" (UID: \"850c3459-5c5f-4b41-9aeb-905c098d4189\") " pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:27:43.917139 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:43.917025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/850c3459-5c5f-4b41-9aeb-905c098d4189-model-cache\") pod \"test-model-76944c-kserve-6bf9bf4b67-6jgxz\" (UID: \"850c3459-5c5f-4b41-9aeb-905c098d4189\") " pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:27:43.917139 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:43.917048    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/850c3459-5c5f-4b41-9aeb-905c098d4189-tmp-dir\") pod \"test-model-76944c-kserve-6bf9bf4b67-6jgxz\" (UID: \"850c3459-5c5f-4b41-9aeb-905c098d4189\") " pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:27:43.917139 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:43.917122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/850c3459-5c5f-4b41-9aeb-905c098d4189-dshm\") pod \"test-model-76944c-kserve-6bf9bf4b67-6jgxz\" (UID: \"850c3459-5c5f-4b41-9aeb-905c098d4189\") " pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:27:44.017688 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:44.017652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/850c3459-5c5f-4b41-9aeb-905c098d4189-tls-certs\") pod \"test-model-76944c-kserve-6bf9bf4b67-6jgxz\" (UID: \"850c3459-5c5f-4b41-9aeb-905c098d4189\") " pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:27:44.017688 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:44.017686    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/850c3459-5c5f-4b41-9aeb-905c098d4189-model-cache\") pod \"test-model-76944c-kserve-6bf9bf4b67-6jgxz\" (UID: \"850c3459-5c5f-4b41-9aeb-905c098d4189\") " pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:27:44.017938 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:44.017705    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/850c3459-5c5f-4b41-9aeb-905c098d4189-tmp-dir\") pod \"test-model-76944c-kserve-6bf9bf4b67-6jgxz\" (UID: \"850c3459-5c5f-4b41-9aeb-905c098d4189\") " pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:27:44.017938 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:44.017728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/850c3459-5c5f-4b41-9aeb-905c098d4189-dshm\") pod \"test-model-76944c-kserve-6bf9bf4b67-6jgxz\" (UID: \"850c3459-5c5f-4b41-9aeb-905c098d4189\") " pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:27:44.017938 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:44.017792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbkm8\" (UniqueName: \"kubernetes.io/projected/850c3459-5c5f-4b41-9aeb-905c098d4189-kube-api-access-xbkm8\") pod \"test-model-76944c-kserve-6bf9bf4b67-6jgxz\" (UID: \"850c3459-5c5f-4b41-9aeb-905c098d4189\") " pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:27:44.017938 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:44.017813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/850c3459-5c5f-4b41-9aeb-905c098d4189-home\") pod \"test-model-76944c-kserve-6bf9bf4b67-6jgxz\" (UID: \"850c3459-5c5f-4b41-9aeb-905c098d4189\") " pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:27:44.018184 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:44.018161    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/850c3459-5c5f-4b41-9aeb-905c098d4189-model-cache\") pod \"test-model-76944c-kserve-6bf9bf4b67-6jgxz\" (UID: \"850c3459-5c5f-4b41-9aeb-905c098d4189\") " pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:27:44.018249 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:44.018193    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/850c3459-5c5f-4b41-9aeb-905c098d4189-tmp-dir\") pod \"test-model-76944c-kserve-6bf9bf4b67-6jgxz\" (UID: \"850c3459-5c5f-4b41-9aeb-905c098d4189\") " pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:27:44.018291 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:44.018267    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/850c3459-5c5f-4b41-9aeb-905c098d4189-home\") pod \"test-model-76944c-kserve-6bf9bf4b67-6jgxz\" (UID: \"850c3459-5c5f-4b41-9aeb-905c098d4189\") " pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:27:44.020158 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:44.020132    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/850c3459-5c5f-4b41-9aeb-905c098d4189-dshm\") pod \"test-model-76944c-kserve-6bf9bf4b67-6jgxz\" (UID: \"850c3459-5c5f-4b41-9aeb-905c098d4189\") " pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:27:44.020324 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:44.020291    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/850c3459-5c5f-4b41-9aeb-905c098d4189-tls-certs\") pod \"test-model-76944c-kserve-6bf9bf4b67-6jgxz\" (UID: \"850c3459-5c5f-4b41-9aeb-905c098d4189\") " pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:27:44.025099 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:44.025081    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbkm8\" (UniqueName: \"kubernetes.io/projected/850c3459-5c5f-4b41-9aeb-905c098d4189-kube-api-access-xbkm8\") pod \"test-model-76944c-kserve-6bf9bf4b67-6jgxz\" (UID: \"850c3459-5c5f-4b41-9aeb-905c098d4189\") " pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:27:44.128831 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:44.128746    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:27:44.403006 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:44.402976    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"]
Sep 17 00:27:44.404290 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:27:44.404264    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod850c3459_5c5f_4b41_9aeb_905c098d4189.slice/crio-8fb91a99e3cd335df7717fa8f08009808a7215dd8f0c3e279204d8cb6bc43fb7 WatchSource:0}: Error finding container 8fb91a99e3cd335df7717fa8f08009808a7215dd8f0c3e279204d8cb6bc43fb7: Status 404 returned error can't find the container with id 8fb91a99e3cd335df7717fa8f08009808a7215dd8f0c3e279204d8cb6bc43fb7
Sep 17 00:27:44.584650 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:44.584599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz" event={"ID":"850c3459-5c5f-4b41-9aeb-905c098d4189","Type":"ContainerStarted","Data":"341ccd84fd72b3ef9d924d77c0ec30d338e27f391c8e98001701cbf7932d74b7"}
Sep 17 00:27:44.584650 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:44.584651    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz" event={"ID":"850c3459-5c5f-4b41-9aeb-905c098d4189","Type":"ContainerStarted","Data":"8fb91a99e3cd335df7717fa8f08009808a7215dd8f0c3e279204d8cb6bc43fb7"}
Sep 17 00:27:44.600752 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:44.600704    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz" podStartSLOduration=1.600689036 podStartE2EDuration="1.600689036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:27:44.599343551 +0000 UTC m=+1077.457836676" watchObservedRunningTime="2026-09-17 00:27:44.600689036 +0000 UTC m=+1077.459182173"
Sep 17 00:27:47.505446 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:47.505400    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6"]
Sep 17 00:27:47.505867 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:47.505658    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6" podUID="d2400633-7a6e-4ca0-8771-77abb2192325" containerName="maas-api" containerID="cri-o://281a9c1ba8907b3c2bf4498883030938f345ca8c683a07a3cd29c350780f79bf" gracePeriod=30
Sep 17 00:27:47.753739 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:47.753711    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6"
Sep 17 00:27:47.854781 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:47.854691    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2400633-7a6e-4ca0-8771-77abb2192325-maas-api-tls\") pod \"d2400633-7a6e-4ca0-8771-77abb2192325\" (UID: \"d2400633-7a6e-4ca0-8771-77abb2192325\") "
Sep 17 00:27:47.854781 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:47.854768    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2400633-7a6e-4ca0-8771-77abb2192325-metrics-certs\") pod \"d2400633-7a6e-4ca0-8771-77abb2192325\" (UID: \"d2400633-7a6e-4ca0-8771-77abb2192325\") "
Sep 17 00:27:47.855022 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:47.854795    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwr4t\" (UniqueName: \"kubernetes.io/projected/d2400633-7a6e-4ca0-8771-77abb2192325-kube-api-access-fwr4t\") pod \"d2400633-7a6e-4ca0-8771-77abb2192325\" (UID: \"d2400633-7a6e-4ca0-8771-77abb2192325\") "
Sep 17 00:27:47.855022 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:47.854838    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2400633-7a6e-4ca0-8771-77abb2192325-openshift-service-ca\") pod \"d2400633-7a6e-4ca0-8771-77abb2192325\" (UID: \"d2400633-7a6e-4ca0-8771-77abb2192325\") "
Sep 17 00:27:47.855343 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:47.855310    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2400633-7a6e-4ca0-8771-77abb2192325-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d2400633-7a6e-4ca0-8771-77abb2192325" (UID: "d2400633-7a6e-4ca0-8771-77abb2192325"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:27:47.857017 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:47.856987    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2400633-7a6e-4ca0-8771-77abb2192325-kube-api-access-fwr4t" (OuterVolumeSpecName: "kube-api-access-fwr4t") pod "d2400633-7a6e-4ca0-8771-77abb2192325" (UID: "d2400633-7a6e-4ca0-8771-77abb2192325"). InnerVolumeSpecName "kube-api-access-fwr4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:27:47.857194 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:47.857168    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2400633-7a6e-4ca0-8771-77abb2192325-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d2400633-7a6e-4ca0-8771-77abb2192325" (UID: "d2400633-7a6e-4ca0-8771-77abb2192325"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:27:47.857408 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:47.857373    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2400633-7a6e-4ca0-8771-77abb2192325-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d2400633-7a6e-4ca0-8771-77abb2192325" (UID: "d2400633-7a6e-4ca0-8771-77abb2192325"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:27:47.956503 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:47.956466    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2400633-7a6e-4ca0-8771-77abb2192325-maas-api-tls\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:27:47.956503 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:47.956495    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2400633-7a6e-4ca0-8771-77abb2192325-metrics-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:27:47.956503 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:47.956504    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwr4t\" (UniqueName: \"kubernetes.io/projected/d2400633-7a6e-4ca0-8771-77abb2192325-kube-api-access-fwr4t\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:27:47.956733 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:47.956514    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2400633-7a6e-4ca0-8771-77abb2192325-openshift-service-ca\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:27:48.608594 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:48.608557    2577 generic.go:356] "Generic (PLEG): container finished" podID="d2400633-7a6e-4ca0-8771-77abb2192325" containerID="281a9c1ba8907b3c2bf4498883030938f345ca8c683a07a3cd29c350780f79bf" exitCode=0
Sep 17 00:27:48.609012 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:48.608600    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6"
Sep 17 00:27:48.609012 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:48.608646    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6" event={"ID":"d2400633-7a6e-4ca0-8771-77abb2192325","Type":"ContainerDied","Data":"281a9c1ba8907b3c2bf4498883030938f345ca8c683a07a3cd29c350780f79bf"}
Sep 17 00:27:48.609012 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:48.608697    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6" event={"ID":"d2400633-7a6e-4ca0-8771-77abb2192325","Type":"ContainerDied","Data":"b77b73cfa37a2399d1aa48451c19384f589a80edb25d8cd1142e76ffa8afba1e"}
Sep 17 00:27:48.609012 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:48.608727    2577 scope.go:117] "RemoveContainer" containerID="281a9c1ba8907b3c2bf4498883030938f345ca8c683a07a3cd29c350780f79bf"
Sep 17 00:27:48.618448 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:48.618423    2577 scope.go:117] "RemoveContainer" containerID="281a9c1ba8907b3c2bf4498883030938f345ca8c683a07a3cd29c350780f79bf"
Sep 17 00:27:48.618742 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:27:48.618724    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281a9c1ba8907b3c2bf4498883030938f345ca8c683a07a3cd29c350780f79bf\": container with ID starting with 281a9c1ba8907b3c2bf4498883030938f345ca8c683a07a3cd29c350780f79bf not found: ID does not exist" containerID="281a9c1ba8907b3c2bf4498883030938f345ca8c683a07a3cd29c350780f79bf"
Sep 17 00:27:48.618809 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:48.618753    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281a9c1ba8907b3c2bf4498883030938f345ca8c683a07a3cd29c350780f79bf"} err="failed to get container status \"281a9c1ba8907b3c2bf4498883030938f345ca8c683a07a3cd29c350780f79bf\": rpc error: code = NotFound desc = could not find container \"281a9c1ba8907b3c2bf4498883030938f345ca8c683a07a3cd29c350780f79bf\": container with ID starting with 281a9c1ba8907b3c2bf4498883030938f345ca8c683a07a3cd29c350780f79bf not found: ID does not exist"
Sep 17 00:27:48.631643 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:48.631621    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6"]
Sep 17 00:27:48.635063 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:48.635040    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-56ff2313-75fc9f875f-tzqb6"]
Sep 17 00:27:48.806061 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:48.806026    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:27:49.607522 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:49.607485    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2400633-7a6e-4ca0-8771-77abb2192325" path="/var/lib/kubelet/pods/d2400633-7a6e-4ca0-8771-77abb2192325/volumes"
Sep 17 00:27:54.129339 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:54.129300    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:27:54.129339 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:54.129351    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:27:54.142293 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:54.142269    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:27:54.652455 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:54.652408    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:27:56.208491 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:56.208461    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28"]
Sep 17 00:27:56.208845 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:56.208801    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2400633-7a6e-4ca0-8771-77abb2192325" containerName="maas-api"
Sep 17 00:27:56.208845 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:56.208813    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2400633-7a6e-4ca0-8771-77abb2192325" containerName="maas-api"
Sep 17 00:27:56.208922 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:56.208886    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2400633-7a6e-4ca0-8771-77abb2192325" containerName="maas-api"
Sep 17 00:27:56.213552 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:56.213516    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28"
Sep 17 00:27:56.216072 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:56.216049    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-8e10e0b5-serving-cert\""
Sep 17 00:27:56.221867 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:56.221461    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28"]
Sep 17 00:27:56.329712 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:56.329671    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a151e2-51e4-4265-aa4f-25a3075f036f-maas-api-tls\") pod \"maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28\" (UID: \"a1a151e2-51e4-4265-aa4f-25a3075f036f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28"
Sep 17 00:27:56.329712 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:56.329713    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w52j\" (UniqueName: \"kubernetes.io/projected/a1a151e2-51e4-4265-aa4f-25a3075f036f-kube-api-access-2w52j\") pod \"maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28\" (UID: \"a1a151e2-51e4-4265-aa4f-25a3075f036f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28"
Sep 17 00:27:56.329932 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:56.329776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a151e2-51e4-4265-aa4f-25a3075f036f-metrics-certs\") pod \"maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28\" (UID: \"a1a151e2-51e4-4265-aa4f-25a3075f036f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28"
Sep 17 00:27:56.329932 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:56.329814    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a151e2-51e4-4265-aa4f-25a3075f036f-openshift-service-ca\") pod \"maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28\" (UID: \"a1a151e2-51e4-4265-aa4f-25a3075f036f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28"
Sep 17 00:27:56.430707 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:56.430670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a151e2-51e4-4265-aa4f-25a3075f036f-openshift-service-ca\") pod \"maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28\" (UID: \"a1a151e2-51e4-4265-aa4f-25a3075f036f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28"
Sep 17 00:27:56.430886 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:56.430742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a151e2-51e4-4265-aa4f-25a3075f036f-maas-api-tls\") pod \"maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28\" (UID: \"a1a151e2-51e4-4265-aa4f-25a3075f036f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28"
Sep 17 00:27:56.430886 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:56.430760    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w52j\" (UniqueName: \"kubernetes.io/projected/a1a151e2-51e4-4265-aa4f-25a3075f036f-kube-api-access-2w52j\") pod \"maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28\" (UID: \"a1a151e2-51e4-4265-aa4f-25a3075f036f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28"
Sep 17 00:27:56.430886 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:56.430790    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a151e2-51e4-4265-aa4f-25a3075f036f-metrics-certs\") pod \"maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28\" (UID: \"a1a151e2-51e4-4265-aa4f-25a3075f036f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28"
Sep 17 00:27:56.431488 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:56.431453    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a151e2-51e4-4265-aa4f-25a3075f036f-openshift-service-ca\") pod \"maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28\" (UID: \"a1a151e2-51e4-4265-aa4f-25a3075f036f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28"
Sep 17 00:27:56.433279 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:56.433250    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a151e2-51e4-4265-aa4f-25a3075f036f-maas-api-tls\") pod \"maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28\" (UID: \"a1a151e2-51e4-4265-aa4f-25a3075f036f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28"
Sep 17 00:27:56.433389 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:56.433365    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a151e2-51e4-4265-aa4f-25a3075f036f-metrics-certs\") pod \"maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28\" (UID: \"a1a151e2-51e4-4265-aa4f-25a3075f036f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28"
Sep 17 00:27:56.440725 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:56.440702    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w52j\" (UniqueName: \"kubernetes.io/projected/a1a151e2-51e4-4265-aa4f-25a3075f036f-kube-api-access-2w52j\") pod \"maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28\" (UID: \"a1a151e2-51e4-4265-aa4f-25a3075f036f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28"
Sep 17 00:27:56.526912 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:56.526824    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28"
Sep 17 00:27:56.805625 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:56.805515    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28"]
Sep 17 00:27:56.807480 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:27:56.807449    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a151e2_51e4_4265_aa4f_25a3075f036f.slice/crio-4501c4041771f2ecb302e86c6018d9c6e11f3e3ae2c3facf8cfe21f57cde8a8d WatchSource:0}: Error finding container 4501c4041771f2ecb302e86c6018d9c6e11f3e3ae2c3facf8cfe21f57cde8a8d: Status 404 returned error can't find the container with id 4501c4041771f2ecb302e86c6018d9c6e11f3e3ae2c3facf8cfe21f57cde8a8d
Sep 17 00:27:57.661904 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:57.661824    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28" event={"ID":"a1a151e2-51e4-4265-aa4f-25a3075f036f","Type":"ContainerStarted","Data":"1de5b6450c8ba5d8591d7b0be6bf8584f33246a328b7e2d232c7bb9d29247dcc"}
Sep 17 00:27:57.661904 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:57.661879    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28" event={"ID":"a1a151e2-51e4-4265-aa4f-25a3075f036f","Type":"ContainerStarted","Data":"4501c4041771f2ecb302e86c6018d9c6e11f3e3ae2c3facf8cfe21f57cde8a8d"}
Sep 17 00:27:57.662284 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:57.661972    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28"
Sep 17 00:27:57.679399 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:27:57.679347    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28" podStartSLOduration=1.182493864 podStartE2EDuration="1.679332607s" totalImagesPullingTime="496.838743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:27:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:27:57.676309976 +0000 UTC m=+1090.534803136" watchObservedRunningTime="2026-09-17 00:27:57.679332607 +0000 UTC m=+1090.537825741"
Sep 17 00:28:03.675372 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:03.675338    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28"
Sep 17 00:28:14.504545 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:14.504506    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:28:33.889106 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:33.889071    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28"]
Sep 17 00:28:33.889644 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:33.889363    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28" podUID="a1a151e2-51e4-4265-aa4f-25a3075f036f" containerName="maas-api" containerID="cri-o://1de5b6450c8ba5d8591d7b0be6bf8584f33246a328b7e2d232c7bb9d29247dcc" gracePeriod=30
Sep 17 00:28:34.138487 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:34.138460    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28"
Sep 17 00:28:34.183903 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:34.183878    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a151e2-51e4-4265-aa4f-25a3075f036f-openshift-service-ca\") pod \"a1a151e2-51e4-4265-aa4f-25a3075f036f\" (UID: \"a1a151e2-51e4-4265-aa4f-25a3075f036f\") "
Sep 17 00:28:34.184045 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:34.183934    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a151e2-51e4-4265-aa4f-25a3075f036f-maas-api-tls\") pod \"a1a151e2-51e4-4265-aa4f-25a3075f036f\" (UID: \"a1a151e2-51e4-4265-aa4f-25a3075f036f\") "
Sep 17 00:28:34.184045 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:34.183993    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a151e2-51e4-4265-aa4f-25a3075f036f-metrics-certs\") pod \"a1a151e2-51e4-4265-aa4f-25a3075f036f\" (UID: \"a1a151e2-51e4-4265-aa4f-25a3075f036f\") "
Sep 17 00:28:34.184045 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:34.184015    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2w52j\" (UniqueName: \"kubernetes.io/projected/a1a151e2-51e4-4265-aa4f-25a3075f036f-kube-api-access-2w52j\") pod \"a1a151e2-51e4-4265-aa4f-25a3075f036f\" (UID: \"a1a151e2-51e4-4265-aa4f-25a3075f036f\") "
Sep 17 00:28:34.184152 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:34.184110    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a151e2-51e4-4265-aa4f-25a3075f036f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a1a151e2-51e4-4265-aa4f-25a3075f036f" (UID: "a1a151e2-51e4-4265-aa4f-25a3075f036f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:28:34.184272 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:34.184258    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a151e2-51e4-4265-aa4f-25a3075f036f-openshift-service-ca\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:28:34.186049 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:34.186014    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a151e2-51e4-4265-aa4f-25a3075f036f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a1a151e2-51e4-4265-aa4f-25a3075f036f" (UID: "a1a151e2-51e4-4265-aa4f-25a3075f036f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:28:34.186153 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:34.186081    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a151e2-51e4-4265-aa4f-25a3075f036f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a1a151e2-51e4-4265-aa4f-25a3075f036f" (UID: "a1a151e2-51e4-4265-aa4f-25a3075f036f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:28:34.186196 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:34.186167    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a151e2-51e4-4265-aa4f-25a3075f036f-kube-api-access-2w52j" (OuterVolumeSpecName: "kube-api-access-2w52j") pod "a1a151e2-51e4-4265-aa4f-25a3075f036f" (UID: "a1a151e2-51e4-4265-aa4f-25a3075f036f"). InnerVolumeSpecName "kube-api-access-2w52j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:28:34.284729 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:34.284694    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1a151e2-51e4-4265-aa4f-25a3075f036f-metrics-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:28:34.284729 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:34.284722    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w52j\" (UniqueName: \"kubernetes.io/projected/a1a151e2-51e4-4265-aa4f-25a3075f036f-kube-api-access-2w52j\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:28:34.284729 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:34.284731    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1a151e2-51e4-4265-aa4f-25a3075f036f-maas-api-tls\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:28:34.876113 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:34.876076    2577 generic.go:356] "Generic (PLEG): container finished" podID="a1a151e2-51e4-4265-aa4f-25a3075f036f" containerID="1de5b6450c8ba5d8591d7b0be6bf8584f33246a328b7e2d232c7bb9d29247dcc" exitCode=0
Sep 17 00:28:34.876323 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:34.876136    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28"
Sep 17 00:28:34.876323 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:34.876162    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28" event={"ID":"a1a151e2-51e4-4265-aa4f-25a3075f036f","Type":"ContainerDied","Data":"1de5b6450c8ba5d8591d7b0be6bf8584f33246a328b7e2d232c7bb9d29247dcc"}
Sep 17 00:28:34.876323 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:34.876224    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28" event={"ID":"a1a151e2-51e4-4265-aa4f-25a3075f036f","Type":"ContainerDied","Data":"4501c4041771f2ecb302e86c6018d9c6e11f3e3ae2c3facf8cfe21f57cde8a8d"}
Sep 17 00:28:34.876323 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:34.876254    2577 scope.go:117] "RemoveContainer" containerID="1de5b6450c8ba5d8591d7b0be6bf8584f33246a328b7e2d232c7bb9d29247dcc"
Sep 17 00:28:34.887994 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:34.887977    2577 scope.go:117] "RemoveContainer" containerID="1de5b6450c8ba5d8591d7b0be6bf8584f33246a328b7e2d232c7bb9d29247dcc"
Sep 17 00:28:34.888290 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:28:34.888268    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de5b6450c8ba5d8591d7b0be6bf8584f33246a328b7e2d232c7bb9d29247dcc\": container with ID starting with 1de5b6450c8ba5d8591d7b0be6bf8584f33246a328b7e2d232c7bb9d29247dcc not found: ID does not exist" containerID="1de5b6450c8ba5d8591d7b0be6bf8584f33246a328b7e2d232c7bb9d29247dcc"
Sep 17 00:28:34.888341 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:34.888299    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de5b6450c8ba5d8591d7b0be6bf8584f33246a328b7e2d232c7bb9d29247dcc"} err="failed to get container status \"1de5b6450c8ba5d8591d7b0be6bf8584f33246a328b7e2d232c7bb9d29247dcc\": rpc error: code = NotFound desc = could not find container \"1de5b6450c8ba5d8591d7b0be6bf8584f33246a328b7e2d232c7bb9d29247dcc\": container with ID starting with 1de5b6450c8ba5d8591d7b0be6bf8584f33246a328b7e2d232c7bb9d29247dcc not found: ID does not exist"
Sep 17 00:28:34.906782 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:34.906758    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28"]
Sep 17 00:28:34.918044 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:34.918020    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8e10e0b5-78fc68965b-69f28"]
Sep 17 00:28:35.607721 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:35.607689    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1a151e2-51e4-4265-aa4f-25a3075f036f" path="/var/lib/kubelet/pods/a1a151e2-51e4-4265-aa4f-25a3075f036f/volumes"
Sep 17 00:28:41.913032 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:41.912994    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:28:45.350751 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:45.350712    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx"]
Sep 17 00:28:45.351263 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:45.350980    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx" podUID="b2fef8e7-d128-4585-8910-983339e7ce25" containerName="maas-api" containerID="cri-o://e5b38896b314f5667e3457585bd085516d05409d4d7f75d5923e78d91afa27de" gracePeriod=30
Sep 17 00:28:45.601533 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:45.601477    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx"
Sep 17 00:28:45.684250 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:45.684211    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2fef8e7-d128-4585-8910-983339e7ce25-maas-api-tls\") pod \"b2fef8e7-d128-4585-8910-983339e7ce25\" (UID: \"b2fef8e7-d128-4585-8910-983339e7ce25\") "
Sep 17 00:28:45.684250 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:45.684253    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2fef8e7-d128-4585-8910-983339e7ce25-metrics-certs\") pod \"b2fef8e7-d128-4585-8910-983339e7ce25\" (UID: \"b2fef8e7-d128-4585-8910-983339e7ce25\") "
Sep 17 00:28:45.684542 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:45.684279    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2fef8e7-d128-4585-8910-983339e7ce25-openshift-service-ca\") pod \"b2fef8e7-d128-4585-8910-983339e7ce25\" (UID: \"b2fef8e7-d128-4585-8910-983339e7ce25\") "
Sep 17 00:28:45.684542 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:45.684337    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rjsw\" (UniqueName: \"kubernetes.io/projected/b2fef8e7-d128-4585-8910-983339e7ce25-kube-api-access-5rjsw\") pod \"b2fef8e7-d128-4585-8910-983339e7ce25\" (UID: \"b2fef8e7-d128-4585-8910-983339e7ce25\") "
Sep 17 00:28:45.684674 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:45.684647    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2fef8e7-d128-4585-8910-983339e7ce25-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b2fef8e7-d128-4585-8910-983339e7ce25" (UID: "b2fef8e7-d128-4585-8910-983339e7ce25"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:28:45.686479 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:45.686457    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2fef8e7-d128-4585-8910-983339e7ce25-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b2fef8e7-d128-4585-8910-983339e7ce25" (UID: "b2fef8e7-d128-4585-8910-983339e7ce25"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:28:45.686907 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:45.686879    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2fef8e7-d128-4585-8910-983339e7ce25-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b2fef8e7-d128-4585-8910-983339e7ce25" (UID: "b2fef8e7-d128-4585-8910-983339e7ce25"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:28:45.686907 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:45.686885    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2fef8e7-d128-4585-8910-983339e7ce25-kube-api-access-5rjsw" (OuterVolumeSpecName: "kube-api-access-5rjsw") pod "b2fef8e7-d128-4585-8910-983339e7ce25" (UID: "b2fef8e7-d128-4585-8910-983339e7ce25"). InnerVolumeSpecName "kube-api-access-5rjsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:28:45.785655 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:45.785611    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2fef8e7-d128-4585-8910-983339e7ce25-maas-api-tls\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:28:45.785655 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:45.785646    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2fef8e7-d128-4585-8910-983339e7ce25-metrics-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:28:45.785655 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:45.785660    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2fef8e7-d128-4585-8910-983339e7ce25-openshift-service-ca\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:28:45.785930 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:45.785676    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rjsw\" (UniqueName: \"kubernetes.io/projected/b2fef8e7-d128-4585-8910-983339e7ce25-kube-api-access-5rjsw\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:28:45.938306 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:45.938271    2577 generic.go:356] "Generic (PLEG): container finished" podID="b2fef8e7-d128-4585-8910-983339e7ce25" containerID="e5b38896b314f5667e3457585bd085516d05409d4d7f75d5923e78d91afa27de" exitCode=0
Sep 17 00:28:45.938498 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:45.938322    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx"
Sep 17 00:28:45.938498 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:45.938352    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx" event={"ID":"b2fef8e7-d128-4585-8910-983339e7ce25","Type":"ContainerDied","Data":"e5b38896b314f5667e3457585bd085516d05409d4d7f75d5923e78d91afa27de"}
Sep 17 00:28:45.938498 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:45.938426    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx" event={"ID":"b2fef8e7-d128-4585-8910-983339e7ce25","Type":"ContainerDied","Data":"3f3f0c8c1654cbfd470877fe3586e043a4eea2a5d30a85f572a1ca83a428d14a"}
Sep 17 00:28:45.938498 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:45.938454    2577 scope.go:117] "RemoveContainer" containerID="e5b38896b314f5667e3457585bd085516d05409d4d7f75d5923e78d91afa27de"
Sep 17 00:28:45.948610 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:45.948582    2577 scope.go:117] "RemoveContainer" containerID="e5b38896b314f5667e3457585bd085516d05409d4d7f75d5923e78d91afa27de"
Sep 17 00:28:45.948935 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:28:45.948908    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5b38896b314f5667e3457585bd085516d05409d4d7f75d5923e78d91afa27de\": container with ID starting with e5b38896b314f5667e3457585bd085516d05409d4d7f75d5923e78d91afa27de not found: ID does not exist" containerID="e5b38896b314f5667e3457585bd085516d05409d4d7f75d5923e78d91afa27de"
Sep 17 00:28:45.949004 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:45.948945    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b38896b314f5667e3457585bd085516d05409d4d7f75d5923e78d91afa27de"} err="failed to get container status \"e5b38896b314f5667e3457585bd085516d05409d4d7f75d5923e78d91afa27de\": rpc error: code = NotFound desc = could not find container \"e5b38896b314f5667e3457585bd085516d05409d4d7f75d5923e78d91afa27de\": container with ID starting with e5b38896b314f5667e3457585bd085516d05409d4d7f75d5923e78d91afa27de not found: ID does not exist"
Sep 17 00:28:45.960565 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:45.960538    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx"]
Sep 17 00:28:45.964288 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:45.964259    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f492a5-75977d9587-rbpwx"]
Sep 17 00:28:47.607218 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:28:47.607181    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2fef8e7-d128-4585-8910-983339e7ce25" path="/var/lib/kubelet/pods/b2fef8e7-d128-4585-8910-983339e7ce25/volumes"
Sep 17 00:29:00.599297 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:00.599235    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:29:42.205680 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:42.205644    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:29:47.625894 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:47.625861    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/ovn-acl-logging/0.log"
Sep 17 00:29:47.628181 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:47.628161    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/ovn-acl-logging/0.log"
Sep 17 00:29:48.394826 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.394791    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"]
Sep 17 00:29:48.395329 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.395299    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2fef8e7-d128-4585-8910-983339e7ce25" containerName="maas-api"
Sep 17 00:29:48.395329 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.395324    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2fef8e7-d128-4585-8910-983339e7ce25" containerName="maas-api"
Sep 17 00:29:48.395537 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.395349    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1a151e2-51e4-4265-aa4f-25a3075f036f" containerName="maas-api"
Sep 17 00:29:48.395537 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.395357    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1a151e2-51e4-4265-aa4f-25a3075f036f" containerName="maas-api"
Sep 17 00:29:48.395537 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.395470    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1a151e2-51e4-4265-aa4f-25a3075f036f" containerName="maas-api"
Sep 17 00:29:48.395537 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.395487    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2fef8e7-d128-4585-8910-983339e7ce25" containerName="maas-api"
Sep 17 00:29:48.398379 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.398363    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:29:48.400894 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.400873    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-cfe049\"/\"premium-w0-cfe049-kserve-self-signed-certs\""
Sep 17 00:29:48.407404 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.407383    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"]
Sep 17 00:29:48.457856 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.457822    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tc7f\" (UniqueName: \"kubernetes.io/projected/0f93433d-4c9e-40e1-8108-410cee61d3eb-kube-api-access-9tc7f\") pod \"premium-w0-cfe049-kserve-85dbc8fc77-wkw4r\" (UID: \"0f93433d-4c9e-40e1-8108-410cee61d3eb\") " pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:29:48.458019 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.457872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f93433d-4c9e-40e1-8108-410cee61d3eb-model-cache\") pod \"premium-w0-cfe049-kserve-85dbc8fc77-wkw4r\" (UID: \"0f93433d-4c9e-40e1-8108-410cee61d3eb\") " pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:29:48.458019 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.457892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f93433d-4c9e-40e1-8108-410cee61d3eb-tmp-dir\") pod \"premium-w0-cfe049-kserve-85dbc8fc77-wkw4r\" (UID: \"0f93433d-4c9e-40e1-8108-410cee61d3eb\") " pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:29:48.458019 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.457911    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f93433d-4c9e-40e1-8108-410cee61d3eb-home\") pod \"premium-w0-cfe049-kserve-85dbc8fc77-wkw4r\" (UID: \"0f93433d-4c9e-40e1-8108-410cee61d3eb\") " pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:29:48.458019 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.457935    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f93433d-4c9e-40e1-8108-410cee61d3eb-dshm\") pod \"premium-w0-cfe049-kserve-85dbc8fc77-wkw4r\" (UID: \"0f93433d-4c9e-40e1-8108-410cee61d3eb\") " pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:29:48.458019 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.457951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f93433d-4c9e-40e1-8108-410cee61d3eb-tls-certs\") pod \"premium-w0-cfe049-kserve-85dbc8fc77-wkw4r\" (UID: \"0f93433d-4c9e-40e1-8108-410cee61d3eb\") " pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:29:48.558840 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.558804    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tc7f\" (UniqueName: \"kubernetes.io/projected/0f93433d-4c9e-40e1-8108-410cee61d3eb-kube-api-access-9tc7f\") pod \"premium-w0-cfe049-kserve-85dbc8fc77-wkw4r\" (UID: \"0f93433d-4c9e-40e1-8108-410cee61d3eb\") " pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:29:48.559044 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.558854    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f93433d-4c9e-40e1-8108-410cee61d3eb-model-cache\") pod \"premium-w0-cfe049-kserve-85dbc8fc77-wkw4r\" (UID: \"0f93433d-4c9e-40e1-8108-410cee61d3eb\") " pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:29:48.559044 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.558880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f93433d-4c9e-40e1-8108-410cee61d3eb-tmp-dir\") pod \"premium-w0-cfe049-kserve-85dbc8fc77-wkw4r\" (UID: \"0f93433d-4c9e-40e1-8108-410cee61d3eb\") " pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:29:48.559044 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.558912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f93433d-4c9e-40e1-8108-410cee61d3eb-home\") pod \"premium-w0-cfe049-kserve-85dbc8fc77-wkw4r\" (UID: \"0f93433d-4c9e-40e1-8108-410cee61d3eb\") " pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:29:48.559044 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.558950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f93433d-4c9e-40e1-8108-410cee61d3eb-dshm\") pod \"premium-w0-cfe049-kserve-85dbc8fc77-wkw4r\" (UID: \"0f93433d-4c9e-40e1-8108-410cee61d3eb\") " pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:29:48.559044 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.558976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f93433d-4c9e-40e1-8108-410cee61d3eb-tls-certs\") pod \"premium-w0-cfe049-kserve-85dbc8fc77-wkw4r\" (UID: \"0f93433d-4c9e-40e1-8108-410cee61d3eb\") " pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:29:48.559352 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.559321    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f93433d-4c9e-40e1-8108-410cee61d3eb-model-cache\") pod \"premium-w0-cfe049-kserve-85dbc8fc77-wkw4r\" (UID: \"0f93433d-4c9e-40e1-8108-410cee61d3eb\") " pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:29:48.559507 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.559358    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f93433d-4c9e-40e1-8108-410cee61d3eb-home\") pod \"premium-w0-cfe049-kserve-85dbc8fc77-wkw4r\" (UID: \"0f93433d-4c9e-40e1-8108-410cee61d3eb\") " pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:29:48.559507 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.559389    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f93433d-4c9e-40e1-8108-410cee61d3eb-tmp-dir\") pod \"premium-w0-cfe049-kserve-85dbc8fc77-wkw4r\" (UID: \"0f93433d-4c9e-40e1-8108-410cee61d3eb\") " pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:29:48.561304 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.561281    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f93433d-4c9e-40e1-8108-410cee61d3eb-dshm\") pod \"premium-w0-cfe049-kserve-85dbc8fc77-wkw4r\" (UID: \"0f93433d-4c9e-40e1-8108-410cee61d3eb\") " pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:29:48.561509 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.561490    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f93433d-4c9e-40e1-8108-410cee61d3eb-tls-certs\") pod \"premium-w0-cfe049-kserve-85dbc8fc77-wkw4r\" (UID: \"0f93433d-4c9e-40e1-8108-410cee61d3eb\") " pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:29:48.566337 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.566309    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tc7f\" (UniqueName: \"kubernetes.io/projected/0f93433d-4c9e-40e1-8108-410cee61d3eb-kube-api-access-9tc7f\") pod \"premium-w0-cfe049-kserve-85dbc8fc77-wkw4r\" (UID: \"0f93433d-4c9e-40e1-8108-410cee61d3eb\") " pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:29:48.708790 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.708754    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:29:48.980914 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:48.980882    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"]
Sep 17 00:29:48.982636 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:29:48.982606    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f93433d_4c9e_40e1_8108_410cee61d3eb.slice/crio-7403ffbf1dd75df6db6d3275dfbec2eb7848ffcc19fb02f55f123d3900025ed1 WatchSource:0}: Error finding container 7403ffbf1dd75df6db6d3275dfbec2eb7848ffcc19fb02f55f123d3900025ed1: Status 404 returned error can't find the container with id 7403ffbf1dd75df6db6d3275dfbec2eb7848ffcc19fb02f55f123d3900025ed1
Sep 17 00:29:49.279324 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:49.279221    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r" event={"ID":"0f93433d-4c9e-40e1-8108-410cee61d3eb","Type":"ContainerStarted","Data":"9b7a108b97fe2af9a0871cb563e76c5ab1ffb25ec64a116b3330bd29d559e4a5"}
Sep 17 00:29:49.279324 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:49.279270    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r" event={"ID":"0f93433d-4c9e-40e1-8108-410cee61d3eb","Type":"ContainerStarted","Data":"7403ffbf1dd75df6db6d3275dfbec2eb7848ffcc19fb02f55f123d3900025ed1"}
Sep 17 00:29:49.295021 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:49.294964    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r" podStartSLOduration=1.294947718 podStartE2EDuration="1.294947718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:29:49.294284091 +0000 UTC m=+1202.152777252" watchObservedRunningTime="2026-09-17 00:29:49.294947718 +0000 UTC m=+1202.153440853"
Sep 17 00:29:58.709505 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:58.709465    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:29:58.709928 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:58.709517    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:29:58.722085 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:58.722054    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:29:59.343164 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:29:59.343128    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:30:00.139408 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:00.139366    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cfe049-29826750-ltqkj"]
Sep 17 00:30:00.142851 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:00.142828    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-76944c-29826750-7gbnw"]
Sep 17 00:30:00.142999 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:00.142944    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cfe049-29826750-ltqkj"
Sep 17 00:30:00.145585 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:00.145554    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jmwpp\""
Sep 17 00:30:00.146664 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:00.146635    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-76944c-29826750-7gbnw"
Sep 17 00:30:00.152320 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:00.152287    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cfe049-29826750-ltqkj"]
Sep 17 00:30:00.157814 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:00.157788    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-76944c-29826750-7gbnw"]
Sep 17 00:30:00.163262 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:00.163240    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cfe049-29826750-ltqkj"
Sep 17 00:30:00.170127 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:00.170104    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-76944c-29826750-7gbnw"
Sep 17 00:30:00.444952 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:00.444887    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cfe049-29826750-ltqkj"]
Sep 17 00:30:00.446348 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:30:00.446298    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c5d985_3d7e_40cd_b53d_9b6c9441eeb1.slice/crio-855ea4064847f0025bbc9905378e9be06a262ac466bd1e413b7f21d2bfb3f881 WatchSource:0}: Error finding container 855ea4064847f0025bbc9905378e9be06a262ac466bd1e413b7f21d2bfb3f881: Status 404 returned error can't find the container with id 855ea4064847f0025bbc9905378e9be06a262ac466bd1e413b7f21d2bfb3f881
Sep 17 00:30:00.447078 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:30:00.447048    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod622b3f7f_3b04_460d_898f_9a8935125a56.slice/crio-9091f1f190f89f054c12035c1a39e4c9bf798386353a7f319bff40e190a4d508 WatchSource:0}: Error finding container 9091f1f190f89f054c12035c1a39e4c9bf798386353a7f319bff40e190a4d508: Status 404 returned error can't find the container with id 9091f1f190f89f054c12035c1a39e4c9bf798386353a7f319bff40e190a4d508
Sep 17 00:30:00.448456 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:00.448427    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-76944c-29826750-7gbnw"]
Sep 17 00:30:01.349979 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:01.349929    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-76944c-29826750-7gbnw" event={"ID":"d0c5d985-3d7e-40cd-b53d-9b6c9441eeb1","Type":"ContainerStarted","Data":"855ea4064847f0025bbc9905378e9be06a262ac466bd1e413b7f21d2bfb3f881"}
Sep 17 00:30:01.351468 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:01.351440    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cfe049-29826750-ltqkj" event={"ID":"622b3f7f-3b04-460d-898f-9a8935125a56","Type":"ContainerStarted","Data":"9091f1f190f89f054c12035c1a39e4c9bf798386353a7f319bff40e190a4d508"}
Sep 17 00:30:04.373713 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:04.373672    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cfe049-29826750-ltqkj" event={"ID":"622b3f7f-3b04-460d-898f-9a8935125a56","Type":"ContainerStarted","Data":"f5411c6aa3200f77781234b68db947dbb432d435aca80b83890207c7b8c107ca"}
Sep 17 00:30:04.375252 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:04.375228    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-76944c-29826750-7gbnw" event={"ID":"d0c5d985-3d7e-40cd-b53d-9b6c9441eeb1","Type":"ContainerStarted","Data":"627b38c35bbabe11289fd8b49da8ef0c8b17d484b78be63d6558d60769ecf0bf"}
Sep 17 00:30:04.387334 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:04.387276    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cfe049-29826750-ltqkj" podStartSLOduration=0.975583549 podStartE2EDuration="4.387260491s" totalImagesPullingTime="3.411676942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:04.386972276 +0000 UTC m=+1217.245465412" watchObservedRunningTime="2026-09-17 00:30:04.387260491 +0000 UTC m=+1217.245753627"
Sep 17 00:30:04.401039 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:04.400996    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-76944c-29826750-7gbnw" podStartSLOduration=0.983413237 podStartE2EDuration="4.400982454s" totalImagesPullingTime="3.417569217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:04.399638008 +0000 UTC m=+1217.258131144" watchObservedRunningTime="2026-09-17 00:30:04.400982454 +0000 UTC m=+1217.259475611"
Sep 17 00:30:05.381877 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:05.381783    2577 generic.go:356] "Generic (PLEG): container finished" podID="622b3f7f-3b04-460d-898f-9a8935125a56" containerID="f5411c6aa3200f77781234b68db947dbb432d435aca80b83890207c7b8c107ca" exitCode=0
Sep 17 00:30:05.382284 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:05.381874    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cfe049-29826750-ltqkj" event={"ID":"622b3f7f-3b04-460d-898f-9a8935125a56","Type":"ContainerDied","Data":"f5411c6aa3200f77781234b68db947dbb432d435aca80b83890207c7b8c107ca"}
Sep 17 00:30:05.383547 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:05.383521    2577 generic.go:356] "Generic (PLEG): container finished" podID="d0c5d985-3d7e-40cd-b53d-9b6c9441eeb1" containerID="627b38c35bbabe11289fd8b49da8ef0c8b17d484b78be63d6558d60769ecf0bf" exitCode=0
Sep 17 00:30:05.383654 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:05.383570    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-76944c-29826750-7gbnw" event={"ID":"d0c5d985-3d7e-40cd-b53d-9b6c9441eeb1","Type":"ContainerDied","Data":"627b38c35bbabe11289fd8b49da8ef0c8b17d484b78be63d6558d60769ecf0bf"}
Sep 17 00:30:06.523866 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:06.523838    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-76944c-29826750-7gbnw"
Sep 17 00:30:06.558374 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:06.558349    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cfe049-29826750-ltqkj"
Sep 17 00:30:07.201788 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:07.201749    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:30:07.395873 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:07.395846    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-76944c-29826750-7gbnw"
Sep 17 00:30:07.395873 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:07.395865    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-76944c-29826750-7gbnw" event={"ID":"d0c5d985-3d7e-40cd-b53d-9b6c9441eeb1","Type":"ContainerDied","Data":"855ea4064847f0025bbc9905378e9be06a262ac466bd1e413b7f21d2bfb3f881"}
Sep 17 00:30:07.396111 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:07.395913    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="855ea4064847f0025bbc9905378e9be06a262ac466bd1e413b7f21d2bfb3f881"
Sep 17 00:30:07.397214 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:07.397192    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cfe049-29826750-ltqkj"
Sep 17 00:30:07.397331 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:07.397225    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cfe049-29826750-ltqkj" event={"ID":"622b3f7f-3b04-460d-898f-9a8935125a56","Type":"ContainerDied","Data":"9091f1f190f89f054c12035c1a39e4c9bf798386353a7f319bff40e190a4d508"}
Sep 17 00:30:07.397331 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:07.397262    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9091f1f190f89f054c12035c1a39e4c9bf798386353a7f319bff40e190a4d508"
Sep 17 00:30:18.895296 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:18.895259    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"]
Sep 17 00:30:18.895903 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:18.895807    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="622b3f7f-3b04-460d-898f-9a8935125a56" containerName="cleanup"
Sep 17 00:30:18.895903 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:18.895829    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="622b3f7f-3b04-460d-898f-9a8935125a56" containerName="cleanup"
Sep 17 00:30:18.895903 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:18.895855    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0c5d985-3d7e-40cd-b53d-9b6c9441eeb1" containerName="cleanup"
Sep 17 00:30:18.895903 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:18.895863    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0c5d985-3d7e-40cd-b53d-9b6c9441eeb1" containerName="cleanup"
Sep 17 00:30:18.896116 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:18.895951    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="622b3f7f-3b04-460d-898f-9a8935125a56" containerName="cleanup"
Sep 17 00:30:18.896116 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:18.895968    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0c5d985-3d7e-40cd-b53d-9b6c9441eeb1" containerName="cleanup"
Sep 17 00:30:18.899007 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:18.898976    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"
Sep 17 00:30:18.901483 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:18.901461    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-44007f60-serving-cert\""
Sep 17 00:30:18.909844 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:18.909813    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"]
Sep 17 00:30:19.026992 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:19.026947    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54d61562-2083-45a3-8c64-d2af14c6d513-metrics-certs\") pod \"maas-api-e2e-mt-44007f60-874d78697-7p92t\" (UID: \"54d61562-2083-45a3-8c64-d2af14c6d513\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"
Sep 17 00:30:19.026992 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:19.026996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54d61562-2083-45a3-8c64-d2af14c6d513-maas-api-tls\") pod \"maas-api-e2e-mt-44007f60-874d78697-7p92t\" (UID: \"54d61562-2083-45a3-8c64-d2af14c6d513\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"
Sep 17 00:30:19.027226 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:19.027023    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d61562-2083-45a3-8c64-d2af14c6d513-openshift-service-ca\") pod \"maas-api-e2e-mt-44007f60-874d78697-7p92t\" (UID: \"54d61562-2083-45a3-8c64-d2af14c6d513\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"
Sep 17 00:30:19.027226 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:19.027038    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwwc6\" (UniqueName: \"kubernetes.io/projected/54d61562-2083-45a3-8c64-d2af14c6d513-kube-api-access-dwwc6\") pod \"maas-api-e2e-mt-44007f60-874d78697-7p92t\" (UID: \"54d61562-2083-45a3-8c64-d2af14c6d513\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"
Sep 17 00:30:19.128146 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:19.128099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54d61562-2083-45a3-8c64-d2af14c6d513-metrics-certs\") pod \"maas-api-e2e-mt-44007f60-874d78697-7p92t\" (UID: \"54d61562-2083-45a3-8c64-d2af14c6d513\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"
Sep 17 00:30:19.128146 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:19.128148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54d61562-2083-45a3-8c64-d2af14c6d513-maas-api-tls\") pod \"maas-api-e2e-mt-44007f60-874d78697-7p92t\" (UID: \"54d61562-2083-45a3-8c64-d2af14c6d513\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"
Sep 17 00:30:19.128443 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:19.128184    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d61562-2083-45a3-8c64-d2af14c6d513-openshift-service-ca\") pod \"maas-api-e2e-mt-44007f60-874d78697-7p92t\" (UID: \"54d61562-2083-45a3-8c64-d2af14c6d513\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"
Sep 17 00:30:19.128443 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:19.128210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwwc6\" (UniqueName: \"kubernetes.io/projected/54d61562-2083-45a3-8c64-d2af14c6d513-kube-api-access-dwwc6\") pod \"maas-api-e2e-mt-44007f60-874d78697-7p92t\" (UID: \"54d61562-2083-45a3-8c64-d2af14c6d513\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"
Sep 17 00:30:19.128443 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:30:19.128261    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-serving-cert: secret "maas-api-e2e-mt-44007f60-serving-cert" not found
Sep 17 00:30:19.128443 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:30:19.128327    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54d61562-2083-45a3-8c64-d2af14c6d513-maas-api-tls podName:54d61562-2083-45a3-8c64-d2af14c6d513 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:19.628311139 +0000 UTC m=+1232.486804251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/54d61562-2083-45a3-8c64-d2af14c6d513-maas-api-tls") pod "maas-api-e2e-mt-44007f60-874d78697-7p92t" (UID: "54d61562-2083-45a3-8c64-d2af14c6d513") : secret "maas-api-e2e-mt-44007f60-serving-cert" not found
Sep 17 00:30:19.129004 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:19.128983    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d61562-2083-45a3-8c64-d2af14c6d513-openshift-service-ca\") pod \"maas-api-e2e-mt-44007f60-874d78697-7p92t\" (UID: \"54d61562-2083-45a3-8c64-d2af14c6d513\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"
Sep 17 00:30:19.130662 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:19.130644    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54d61562-2083-45a3-8c64-d2af14c6d513-metrics-certs\") pod \"maas-api-e2e-mt-44007f60-874d78697-7p92t\" (UID: \"54d61562-2083-45a3-8c64-d2af14c6d513\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"
Sep 17 00:30:19.137927 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:19.137901    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwwc6\" (UniqueName: \"kubernetes.io/projected/54d61562-2083-45a3-8c64-d2af14c6d513-kube-api-access-dwwc6\") pod \"maas-api-e2e-mt-44007f60-874d78697-7p92t\" (UID: \"54d61562-2083-45a3-8c64-d2af14c6d513\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"
Sep 17 00:30:19.632705 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:19.632663    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54d61562-2083-45a3-8c64-d2af14c6d513-maas-api-tls\") pod \"maas-api-e2e-mt-44007f60-874d78697-7p92t\" (UID: \"54d61562-2083-45a3-8c64-d2af14c6d513\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"
Sep 17 00:30:19.635157 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:19.635125    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54d61562-2083-45a3-8c64-d2af14c6d513-maas-api-tls\") pod \"maas-api-e2e-mt-44007f60-874d78697-7p92t\" (UID: \"54d61562-2083-45a3-8c64-d2af14c6d513\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"
Sep 17 00:30:19.812773 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:19.812736    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"
Sep 17 00:30:20.088165 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:20.088111    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"]
Sep 17 00:30:20.091147 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:30:20.091118    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d61562_2083_45a3_8c64_d2af14c6d513.slice/crio-90dbcfe129d635dc8da7bd216ddaefeb46148e13b54d7cb18c26f1c598d8c9cb WatchSource:0}: Error finding container 90dbcfe129d635dc8da7bd216ddaefeb46148e13b54d7cb18c26f1c598d8c9cb: Status 404 returned error can't find the container with id 90dbcfe129d635dc8da7bd216ddaefeb46148e13b54d7cb18c26f1c598d8c9cb
Sep 17 00:30:20.476207 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:20.476175    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t" event={"ID":"54d61562-2083-45a3-8c64-d2af14c6d513","Type":"ContainerStarted","Data":"90dbcfe129d635dc8da7bd216ddaefeb46148e13b54d7cb18c26f1c598d8c9cb"}
Sep 17 00:30:21.484467 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:21.484399    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t" event={"ID":"54d61562-2083-45a3-8c64-d2af14c6d513","Type":"ContainerStarted","Data":"88c61d48269a1bfa43884bce6921cf6f10f6171bf1efb1a51277e37a83a96ea4"}
Sep 17 00:30:21.484915 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:21.484512    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"
Sep 17 00:30:21.503498 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:21.503442    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t" podStartSLOduration=3.079546921 podStartE2EDuration="3.503409972s" totalImagesPullingTime="423.863051ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:21.500836371 +0000 UTC m=+1234.359329507" watchObservedRunningTime="2026-09-17 00:30:21.503409972 +0000 UTC m=+1234.361903106"
Sep 17 00:30:27.496429 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:27.496390    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"
Sep 17 00:30:33.707997 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:33.707957    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:30:42.801256 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.801173    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"]
Sep 17 00:30:42.805863 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.805839    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:30:42.809306 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.809278    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-abe86c\"/\"default-dockercfg-dhfl4\""
Sep 17 00:30:42.809470 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.809322    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-abe86c\"/\"kube-root-ca.crt\""
Sep 17 00:30:42.809470 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.809341    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-abe86c\"/\"openshift-service-ca.crt\""
Sep 17 00:30:42.809646 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.809467    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-abe86c\"/\"explicit-model-abe86c-kserve-self-signed-certs\""
Sep 17 00:30:42.812368 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.812343    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"]
Sep 17 00:30:42.867974 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.867935    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-dshm\") pod \"explicit-model-abe86c-kserve-7c9d7b9c9-jl49w\" (UID: \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\") " pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:30:42.868162 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.868005    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-home\") pod \"explicit-model-abe86c-kserve-7c9d7b9c9-jl49w\" (UID: \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\") " pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:30:42.868162 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.868024    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-model-cache\") pod \"explicit-model-abe86c-kserve-7c9d7b9c9-jl49w\" (UID: \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\") " pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:30:42.868162 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.868052    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-tls-certs\") pod \"explicit-model-abe86c-kserve-7c9d7b9c9-jl49w\" (UID: \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\") " pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:30:42.868333 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.868157    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-tmp-dir\") pod \"explicit-model-abe86c-kserve-7c9d7b9c9-jl49w\" (UID: \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\") " pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:30:42.868333 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.868201    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44mgh\" (UniqueName: \"kubernetes.io/projected/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-kube-api-access-44mgh\") pod \"explicit-model-abe86c-kserve-7c9d7b9c9-jl49w\" (UID: \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\") " pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:30:42.969032 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.968988    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-home\") pod \"explicit-model-abe86c-kserve-7c9d7b9c9-jl49w\" (UID: \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\") " pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:30:42.969032 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.969032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-model-cache\") pod \"explicit-model-abe86c-kserve-7c9d7b9c9-jl49w\" (UID: \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\") " pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:30:42.969265 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.969067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-tls-certs\") pod \"explicit-model-abe86c-kserve-7c9d7b9c9-jl49w\" (UID: \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\") " pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:30:42.969265 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.969120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-tmp-dir\") pod \"explicit-model-abe86c-kserve-7c9d7b9c9-jl49w\" (UID: \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\") " pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:30:42.969265 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.969152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44mgh\" (UniqueName: \"kubernetes.io/projected/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-kube-api-access-44mgh\") pod \"explicit-model-abe86c-kserve-7c9d7b9c9-jl49w\" (UID: \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\") " pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:30:42.969265 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.969193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-dshm\") pod \"explicit-model-abe86c-kserve-7c9d7b9c9-jl49w\" (UID: \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\") " pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:30:42.969533 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.969512    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-home\") pod \"explicit-model-abe86c-kserve-7c9d7b9c9-jl49w\" (UID: \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\") " pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:30:42.969589 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.969566    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-model-cache\") pod \"explicit-model-abe86c-kserve-7c9d7b9c9-jl49w\" (UID: \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\") " pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:30:42.969631 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.969571    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-tmp-dir\") pod \"explicit-model-abe86c-kserve-7c9d7b9c9-jl49w\" (UID: \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\") " pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:30:42.971495 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.971464    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-dshm\") pod \"explicit-model-abe86c-kserve-7c9d7b9c9-jl49w\" (UID: \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\") " pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:30:42.971742 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.971727    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-tls-certs\") pod \"explicit-model-abe86c-kserve-7c9d7b9c9-jl49w\" (UID: \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\") " pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:30:42.994374 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:42.994339    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44mgh\" (UniqueName: \"kubernetes.io/projected/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-kube-api-access-44mgh\") pod \"explicit-model-abe86c-kserve-7c9d7b9c9-jl49w\" (UID: \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\") " pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:30:43.116934 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:43.116839    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:30:43.390241 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:43.390207    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"]
Sep 17 00:30:43.391963 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:30:43.391932    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaee2cc62_b21b_41d3_8ffe_79d4ece6f289.slice/crio-95e727c507d7ee649d3fc1fce0649ac048bcfca49e81906a53ce8bd7d0a6c14c WatchSource:0}: Error finding container 95e727c507d7ee649d3fc1fce0649ac048bcfca49e81906a53ce8bd7d0a6c14c: Status 404 returned error can't find the container with id 95e727c507d7ee649d3fc1fce0649ac048bcfca49e81906a53ce8bd7d0a6c14c
Sep 17 00:30:43.608079 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:43.608040    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w" event={"ID":"aee2cc62-b21b-41d3-8ffe-79d4ece6f289","Type":"ContainerStarted","Data":"7c8f539b70493efcb9ec11b7b22ff02d4ce79ca53465216761a4ccb80b13726f"}
Sep 17 00:30:43.608079 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:43.608081    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w" event={"ID":"aee2cc62-b21b-41d3-8ffe-79d4ece6f289","Type":"ContainerStarted","Data":"95e727c507d7ee649d3fc1fce0649ac048bcfca49e81906a53ce8bd7d0a6c14c"}
Sep 17 00:30:43.631724 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:43.631669    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w" podStartSLOduration=1.631653203 podStartE2EDuration="1.631653203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:43.629182468 +0000 UTC m=+1256.487675616" watchObservedRunningTime="2026-09-17 00:30:43.631653203 +0000 UTC m=+1256.490146340"
Sep 17 00:30:53.117099 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:53.117053    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:30:53.117756 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:53.117146    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:30:53.130117 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:53.130086    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:30:53.674284 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:30:53.674250    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:31:03.507249 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:03.507207    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"]
Sep 17 00:31:03.507691 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:03.507502    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t" podUID="54d61562-2083-45a3-8c64-d2af14c6d513" containerName="maas-api" containerID="cri-o://88c61d48269a1bfa43884bce6921cf6f10f6171bf1efb1a51277e37a83a96ea4" gracePeriod=30
Sep 17 00:31:03.726452 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:03.726398    2577 generic.go:356] "Generic (PLEG): container finished" podID="54d61562-2083-45a3-8c64-d2af14c6d513" containerID="88c61d48269a1bfa43884bce6921cf6f10f6171bf1efb1a51277e37a83a96ea4" exitCode=0
Sep 17 00:31:03.726679 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:03.726479    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t" event={"ID":"54d61562-2083-45a3-8c64-d2af14c6d513","Type":"ContainerDied","Data":"88c61d48269a1bfa43884bce6921cf6f10f6171bf1efb1a51277e37a83a96ea4"}
Sep 17 00:31:03.759393 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:03.759329    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"
Sep 17 00:31:03.864488 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:03.864458    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d61562-2083-45a3-8c64-d2af14c6d513-openshift-service-ca\") pod \"54d61562-2083-45a3-8c64-d2af14c6d513\" (UID: \"54d61562-2083-45a3-8c64-d2af14c6d513\") "
Sep 17 00:31:03.864682 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:03.864592    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54d61562-2083-45a3-8c64-d2af14c6d513-metrics-certs\") pod \"54d61562-2083-45a3-8c64-d2af14c6d513\" (UID: \"54d61562-2083-45a3-8c64-d2af14c6d513\") "
Sep 17 00:31:03.864682 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:03.864636    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54d61562-2083-45a3-8c64-d2af14c6d513-maas-api-tls\") pod \"54d61562-2083-45a3-8c64-d2af14c6d513\" (UID: \"54d61562-2083-45a3-8c64-d2af14c6d513\") "
Sep 17 00:31:03.864682 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:03.864657    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwwc6\" (UniqueName: \"kubernetes.io/projected/54d61562-2083-45a3-8c64-d2af14c6d513-kube-api-access-dwwc6\") pod \"54d61562-2083-45a3-8c64-d2af14c6d513\" (UID: \"54d61562-2083-45a3-8c64-d2af14c6d513\") "
Sep 17 00:31:03.864919 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:03.864886    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54d61562-2083-45a3-8c64-d2af14c6d513-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "54d61562-2083-45a3-8c64-d2af14c6d513" (UID: "54d61562-2083-45a3-8c64-d2af14c6d513"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:31:03.866762 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:03.866726    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54d61562-2083-45a3-8c64-d2af14c6d513-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "54d61562-2083-45a3-8c64-d2af14c6d513" (UID: "54d61562-2083-45a3-8c64-d2af14c6d513"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:31:03.866927 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:03.866902    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d61562-2083-45a3-8c64-d2af14c6d513-kube-api-access-dwwc6" (OuterVolumeSpecName: "kube-api-access-dwwc6") pod "54d61562-2083-45a3-8c64-d2af14c6d513" (UID: "54d61562-2083-45a3-8c64-d2af14c6d513"). InnerVolumeSpecName "kube-api-access-dwwc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:31:03.867002 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:03.866909    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54d61562-2083-45a3-8c64-d2af14c6d513-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "54d61562-2083-45a3-8c64-d2af14c6d513" (UID: "54d61562-2083-45a3-8c64-d2af14c6d513"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:31:03.965775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:03.965719    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54d61562-2083-45a3-8c64-d2af14c6d513-metrics-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:31:03.965775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:03.965769    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54d61562-2083-45a3-8c64-d2af14c6d513-maas-api-tls\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:31:03.965775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:03.965780    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwwc6\" (UniqueName: \"kubernetes.io/projected/54d61562-2083-45a3-8c64-d2af14c6d513-kube-api-access-dwwc6\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:31:03.965775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:03.965791    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d61562-2083-45a3-8c64-d2af14c6d513-openshift-service-ca\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:31:04.733612 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:04.733577    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"
Sep 17 00:31:04.734086 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:04.733624    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t" event={"ID":"54d61562-2083-45a3-8c64-d2af14c6d513","Type":"ContainerDied","Data":"90dbcfe129d635dc8da7bd216ddaefeb46148e13b54d7cb18c26f1c598d8c9cb"}
Sep 17 00:31:04.734086 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:04.733705    2577 scope.go:117] "RemoveContainer" containerID="88c61d48269a1bfa43884bce6921cf6f10f6171bf1efb1a51277e37a83a96ea4"
Sep 17 00:31:04.758917 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:04.758880    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"]
Sep 17 00:31:04.764041 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:04.764006    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-44007f60-874d78697-7p92t"]
Sep 17 00:31:05.608391 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:05.608355    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54d61562-2083-45a3-8c64-d2af14c6d513" path="/var/lib/kubelet/pods/54d61562-2083-45a3-8c64-d2af14c6d513/volumes"
Sep 17 00:31:09.704030 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:09.703988    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:31:29.808839 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:29.808797    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r"]
Sep 17 00:31:29.809387 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:29.809321    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54d61562-2083-45a3-8c64-d2af14c6d513" containerName="maas-api"
Sep 17 00:31:29.809387 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:29.809344    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="54d61562-2083-45a3-8c64-d2af14c6d513" containerName="maas-api"
Sep 17 00:31:29.809572 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:29.809490    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="54d61562-2083-45a3-8c64-d2af14c6d513" containerName="maas-api"
Sep 17 00:31:29.811536 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:29.811512    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r"
Sep 17 00:31:29.813793 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:29.813772    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-48e479-serving-cert\""
Sep 17 00:31:29.823791 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:29.823765    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r"]
Sep 17 00:31:29.899255 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:29.899218    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpc8g\" (UniqueName: \"kubernetes.io/projected/b86d40c8-5566-45fe-a649-faf8a7231e95-kube-api-access-dpc8g\") pod \"maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r\" (UID: \"b86d40c8-5566-45fe-a649-faf8a7231e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r"
Sep 17 00:31:29.899464 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:29.899321    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b86d40c8-5566-45fe-a649-faf8a7231e95-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r\" (UID: \"b86d40c8-5566-45fe-a649-faf8a7231e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r"
Sep 17 00:31:29.899464 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:29.899353    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b86d40c8-5566-45fe-a649-faf8a7231e95-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r\" (UID: \"b86d40c8-5566-45fe-a649-faf8a7231e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r"
Sep 17 00:31:29.899464 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:29.899398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86d40c8-5566-45fe-a649-faf8a7231e95-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r\" (UID: \"b86d40c8-5566-45fe-a649-faf8a7231e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r"
Sep 17 00:31:30.000580 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:30.000539    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b86d40c8-5566-45fe-a649-faf8a7231e95-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r\" (UID: \"b86d40c8-5566-45fe-a649-faf8a7231e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r"
Sep 17 00:31:30.000580 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:30.000586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b86d40c8-5566-45fe-a649-faf8a7231e95-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r\" (UID: \"b86d40c8-5566-45fe-a649-faf8a7231e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r"
Sep 17 00:31:30.000865 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:30.000603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86d40c8-5566-45fe-a649-faf8a7231e95-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r\" (UID: \"b86d40c8-5566-45fe-a649-faf8a7231e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r"
Sep 17 00:31:30.000865 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:30.000793    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpc8g\" (UniqueName: \"kubernetes.io/projected/b86d40c8-5566-45fe-a649-faf8a7231e95-kube-api-access-dpc8g\") pod \"maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r\" (UID: \"b86d40c8-5566-45fe-a649-faf8a7231e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r"
Sep 17 00:31:30.001336 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:30.001311    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b86d40c8-5566-45fe-a649-faf8a7231e95-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r\" (UID: \"b86d40c8-5566-45fe-a649-faf8a7231e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r"
Sep 17 00:31:30.003684 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:30.003664    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b86d40c8-5566-45fe-a649-faf8a7231e95-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r\" (UID: \"b86d40c8-5566-45fe-a649-faf8a7231e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r"
Sep 17 00:31:30.003834 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:30.003812    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86d40c8-5566-45fe-a649-faf8a7231e95-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r\" (UID: \"b86d40c8-5566-45fe-a649-faf8a7231e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r"
Sep 17 00:31:30.008893 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:30.008867    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpc8g\" (UniqueName: \"kubernetes.io/projected/b86d40c8-5566-45fe-a649-faf8a7231e95-kube-api-access-dpc8g\") pod \"maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r\" (UID: \"b86d40c8-5566-45fe-a649-faf8a7231e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r"
Sep 17 00:31:30.123672 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:30.123576    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r"
Sep 17 00:31:30.400716 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:30.400687    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r"]
Sep 17 00:31:30.402631 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:31:30.402599    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86d40c8_5566_45fe_a649_faf8a7231e95.slice/crio-616c12977c87c4a6fa053aadbe172d7f594293b743c92cbf06cce4dca554de31 WatchSource:0}: Error finding container 616c12977c87c4a6fa053aadbe172d7f594293b743c92cbf06cce4dca554de31: Status 404 returned error can't find the container with id 616c12977c87c4a6fa053aadbe172d7f594293b743c92cbf06cce4dca554de31
Sep 17 00:31:30.892328 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:30.892282    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r" event={"ID":"b86d40c8-5566-45fe-a649-faf8a7231e95","Type":"ContainerStarted","Data":"3f057d2d7a6002c5b660cc69974c6cc683015904a1475331fea850e26d661c44"}
Sep 17 00:31:30.892755 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:30.892345    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r"
Sep 17 00:31:30.892755 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:30.892363    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r" event={"ID":"b86d40c8-5566-45fe-a649-faf8a7231e95","Type":"ContainerStarted","Data":"616c12977c87c4a6fa053aadbe172d7f594293b743c92cbf06cce4dca554de31"}
Sep 17 00:31:30.917453 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:30.917376    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r" podStartSLOduration=1.5607460020000001 podStartE2EDuration="1.917357421s" totalImagesPullingTime="356.611419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:31:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:31:30.914682084 +0000 UTC m=+1303.773175220" watchObservedRunningTime="2026-09-17 00:31:30.917357421 +0000 UTC m=+1303.775850586"
Sep 17 00:31:35.111201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:35.111162    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"]
Sep 17 00:31:36.903811 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:36.903778    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r"
Sep 17 00:31:38.306196 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:38.306155    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:31:50.130923 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:50.130876    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w" podUID="aee2cc62-b21b-41d3-8ffe-79d4ece6f289" containerName="main" containerID="cri-o://7c8f539b70493efcb9ec11b7b22ff02d4ce79ca53465216761a4ccb80b13726f" gracePeriod=45
Sep 17 00:31:50.387923 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:50.387858    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:31:50.505023 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:50.504975    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-tmp-dir\") pod \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\" (UID: \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\") "
Sep 17 00:31:50.505224 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:50.505075    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-home\") pod \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\" (UID: \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\") "
Sep 17 00:31:50.505224 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:50.505103    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-dshm\") pod \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\" (UID: \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\") "
Sep 17 00:31:50.505224 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:50.505178    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-model-cache\") pod \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\" (UID: \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\") "
Sep 17 00:31:50.505224 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:50.505212    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-tls-certs\") pod \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\" (UID: \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\") "
Sep 17 00:31:50.505518 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:50.505250    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44mgh\" (UniqueName: \"kubernetes.io/projected/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-kube-api-access-44mgh\") pod \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\" (UID: \"aee2cc62-b21b-41d3-8ffe-79d4ece6f289\") "
Sep 17 00:31:50.505518 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:50.505275    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aee2cc62-b21b-41d3-8ffe-79d4ece6f289" (UID: "aee2cc62-b21b-41d3-8ffe-79d4ece6f289"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:31:50.505518 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:50.505316    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-home" (OuterVolumeSpecName: "home") pod "aee2cc62-b21b-41d3-8ffe-79d4ece6f289" (UID: "aee2cc62-b21b-41d3-8ffe-79d4ece6f289"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:31:50.505518 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:50.505449    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-model-cache" (OuterVolumeSpecName: "model-cache") pod "aee2cc62-b21b-41d3-8ffe-79d4ece6f289" (UID: "aee2cc62-b21b-41d3-8ffe-79d4ece6f289"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:31:50.505659 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:50.505544    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-model-cache\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:31:50.505659 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:50.505556    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-tmp-dir\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:31:50.505659 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:50.505565    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-home\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:31:50.507585 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:50.507561    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-kube-api-access-44mgh" (OuterVolumeSpecName: "kube-api-access-44mgh") pod "aee2cc62-b21b-41d3-8ffe-79d4ece6f289" (UID: "aee2cc62-b21b-41d3-8ffe-79d4ece6f289"). InnerVolumeSpecName "kube-api-access-44mgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:31:50.507680 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:50.507591    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-dshm" (OuterVolumeSpecName: "dshm") pod "aee2cc62-b21b-41d3-8ffe-79d4ece6f289" (UID: "aee2cc62-b21b-41d3-8ffe-79d4ece6f289"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:31:50.507680 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:50.507596    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aee2cc62-b21b-41d3-8ffe-79d4ece6f289" (UID: "aee2cc62-b21b-41d3-8ffe-79d4ece6f289"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:31:50.606738 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:50.606686    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-tls-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:31:50.606738 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:50.606729    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44mgh\" (UniqueName: \"kubernetes.io/projected/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-kube-api-access-44mgh\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:31:50.606956 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:50.606752    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aee2cc62-b21b-41d3-8ffe-79d4ece6f289-dshm\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:31:51.006638 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:51.006596    2577 generic.go:356] "Generic (PLEG): container finished" podID="aee2cc62-b21b-41d3-8ffe-79d4ece6f289" containerID="7c8f539b70493efcb9ec11b7b22ff02d4ce79ca53465216761a4ccb80b13726f" exitCode=0
Sep 17 00:31:51.006825 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:51.006663    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"
Sep 17 00:31:51.006825 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:51.006686    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w" event={"ID":"aee2cc62-b21b-41d3-8ffe-79d4ece6f289","Type":"ContainerDied","Data":"7c8f539b70493efcb9ec11b7b22ff02d4ce79ca53465216761a4ccb80b13726f"}
Sep 17 00:31:51.006825 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:51.006745    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w" event={"ID":"aee2cc62-b21b-41d3-8ffe-79d4ece6f289","Type":"ContainerDied","Data":"95e727c507d7ee649d3fc1fce0649ac048bcfca49e81906a53ce8bd7d0a6c14c"}
Sep 17 00:31:51.006825 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:51.006763    2577 scope.go:117] "RemoveContainer" containerID="7c8f539b70493efcb9ec11b7b22ff02d4ce79ca53465216761a4ccb80b13726f"
Sep 17 00:31:51.016988 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:51.016970    2577 scope.go:117] "RemoveContainer" containerID="7c8f539b70493efcb9ec11b7b22ff02d4ce79ca53465216761a4ccb80b13726f"
Sep 17 00:31:51.017283 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:31:51.017258    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8f539b70493efcb9ec11b7b22ff02d4ce79ca53465216761a4ccb80b13726f\": container with ID starting with 7c8f539b70493efcb9ec11b7b22ff02d4ce79ca53465216761a4ccb80b13726f not found: ID does not exist" containerID="7c8f539b70493efcb9ec11b7b22ff02d4ce79ca53465216761a4ccb80b13726f"
Sep 17 00:31:51.017355 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:51.017297    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8f539b70493efcb9ec11b7b22ff02d4ce79ca53465216761a4ccb80b13726f"} err="failed to get container status \"7c8f539b70493efcb9ec11b7b22ff02d4ce79ca53465216761a4ccb80b13726f\": rpc error: code = NotFound desc = could not find container \"7c8f539b70493efcb9ec11b7b22ff02d4ce79ca53465216761a4ccb80b13726f\": container with ID starting with 7c8f539b70493efcb9ec11b7b22ff02d4ce79ca53465216761a4ccb80b13726f not found: ID does not exist"
Sep 17 00:31:51.028359 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:51.028330    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"]
Sep 17 00:31:51.030629 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:51.030608    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-abe86c/explicit-model-abe86c-kserve-7c9d7b9c9-jl49w"]
Sep 17 00:31:51.608490 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:31:51.608445    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aee2cc62-b21b-41d3-8ffe-79d4ece6f289" path="/var/lib/kubelet/pods/aee2cc62-b21b-41d3-8ffe-79d4ece6f289/volumes"
Sep 17 00:32:00.689924 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.689878    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"]
Sep 17 00:32:00.690566 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.690409    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aee2cc62-b21b-41d3-8ffe-79d4ece6f289" containerName="main"
Sep 17 00:32:00.690566 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.690446    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aee2cc62-b21b-41d3-8ffe-79d4ece6f289" containerName="main"
Sep 17 00:32:00.690698 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.690577    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="aee2cc62-b21b-41d3-8ffe-79d4ece6f289" containerName="main"
Sep 17 00:32:00.693184 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.693161    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:32:00.695612 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.695577    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-cfe049\"/\"distinct-w0-cfe049-kserve-self-signed-certs\""
Sep 17 00:32:00.703576 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.703554    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"]
Sep 17 00:32:00.807013 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.806960    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-home\") pod \"distinct-w0-cfe049-kserve-5db7d67668-hpvgf\" (UID: \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\") " pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:32:00.807013 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.807012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-dshm\") pod \"distinct-w0-cfe049-kserve-5db7d67668-hpvgf\" (UID: \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\") " pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:32:00.807284 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.807086    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm2r6\" (UniqueName: \"kubernetes.io/projected/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-kube-api-access-zm2r6\") pod \"distinct-w0-cfe049-kserve-5db7d67668-hpvgf\" (UID: \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\") " pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:32:00.807284 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.807154    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-model-cache\") pod \"distinct-w0-cfe049-kserve-5db7d67668-hpvgf\" (UID: \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\") " pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:32:00.807284 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.807177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-tls-certs\") pod \"distinct-w0-cfe049-kserve-5db7d67668-hpvgf\" (UID: \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\") " pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:32:00.807284 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.807240    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-tmp-dir\") pod \"distinct-w0-cfe049-kserve-5db7d67668-hpvgf\" (UID: \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\") " pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:32:00.908496 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.908448    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-model-cache\") pod \"distinct-w0-cfe049-kserve-5db7d67668-hpvgf\" (UID: \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\") " pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:32:00.908496 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.908496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-tls-certs\") pod \"distinct-w0-cfe049-kserve-5db7d67668-hpvgf\" (UID: \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\") " pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:32:00.908775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.908543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-tmp-dir\") pod \"distinct-w0-cfe049-kserve-5db7d67668-hpvgf\" (UID: \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\") " pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:32:00.908775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.908572    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-home\") pod \"distinct-w0-cfe049-kserve-5db7d67668-hpvgf\" (UID: \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\") " pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:32:00.908775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.908599    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-dshm\") pod \"distinct-w0-cfe049-kserve-5db7d67668-hpvgf\" (UID: \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\") " pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:32:00.908775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.908653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm2r6\" (UniqueName: \"kubernetes.io/projected/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-kube-api-access-zm2r6\") pod \"distinct-w0-cfe049-kserve-5db7d67668-hpvgf\" (UID: \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\") " pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:32:00.908995 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.908887    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-model-cache\") pod \"distinct-w0-cfe049-kserve-5db7d67668-hpvgf\" (UID: \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\") " pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:32:00.908995 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.908943    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-tmp-dir\") pod \"distinct-w0-cfe049-kserve-5db7d67668-hpvgf\" (UID: \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\") " pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:32:00.908995 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.908959    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-home\") pod \"distinct-w0-cfe049-kserve-5db7d67668-hpvgf\" (UID: \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\") " pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:32:00.910855 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.910832    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-dshm\") pod \"distinct-w0-cfe049-kserve-5db7d67668-hpvgf\" (UID: \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\") " pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:32:00.911182 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.911151    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-tls-certs\") pod \"distinct-w0-cfe049-kserve-5db7d67668-hpvgf\" (UID: \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\") " pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:32:00.918585 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:00.918556    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm2r6\" (UniqueName: \"kubernetes.io/projected/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-kube-api-access-zm2r6\") pod \"distinct-w0-cfe049-kserve-5db7d67668-hpvgf\" (UID: \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\") " pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:32:01.002911 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:01.002811    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:32:01.278539 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:01.278513    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"]
Sep 17 00:32:01.280760 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:32:01.280726    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2455bef_2bec_4e2e_9aed_f3734c47b8c9.slice/crio-b3de551273098f31fcba10820fe4c5c1114393bbbe5fb3e0652558dc91975b6a WatchSource:0}: Error finding container b3de551273098f31fcba10820fe4c5c1114393bbbe5fb3e0652558dc91975b6a: Status 404 returned error can't find the container with id b3de551273098f31fcba10820fe4c5c1114393bbbe5fb3e0652558dc91975b6a
Sep 17 00:32:02.079933 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:02.079889    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf" event={"ID":"a2455bef-2bec-4e2e-9aed-f3734c47b8c9","Type":"ContainerStarted","Data":"d0e41ead288ee83216691e320f8af73ac8812d00a6452faf59845111921d8595"}
Sep 17 00:32:02.080453 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:02.079946    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf" event={"ID":"a2455bef-2bec-4e2e-9aed-f3734c47b8c9","Type":"ContainerStarted","Data":"b3de551273098f31fcba10820fe4c5c1114393bbbe5fb3e0652558dc91975b6a"}
Sep 17 00:32:02.097269 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:02.097205    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf" podStartSLOduration=2.09718953 podStartE2EDuration="2.09718953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:32:02.094833185 +0000 UTC m=+1334.953326321" watchObservedRunningTime="2026-09-17 00:32:02.09718953 +0000 UTC m=+1334.955682665"
Sep 17 00:32:06.500617 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:06.500577    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:32:11.003798 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:11.003689    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:32:11.003798 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:11.003755    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:32:11.016525 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:11.016490    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:32:11.141812 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:11.141779    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:32:31.132087 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:31.132050    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7828-7a3e16"]
Sep 17 00:32:31.135551 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:31.135534    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7828-7a3e16"
Sep 17 00:32:31.138162 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:31.138135    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hf767\""
Sep 17 00:32:31.141833 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:31.141804    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7828-7a3e16"]
Sep 17 00:32:31.206999 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:31.206955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cqh9\" (UniqueName: \"kubernetes.io/projected/e42866cc-5509-4428-9434-cb71a2b59e91-kube-api-access-2cqh9\") pod \"test-curl-7828-7a3e16\" (UID: \"e42866cc-5509-4428-9434-cb71a2b59e91\") " pod="opendatahub/test-curl-7828-7a3e16"
Sep 17 00:32:31.308216 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:31.308167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cqh9\" (UniqueName: \"kubernetes.io/projected/e42866cc-5509-4428-9434-cb71a2b59e91-kube-api-access-2cqh9\") pod \"test-curl-7828-7a3e16\" (UID: \"e42866cc-5509-4428-9434-cb71a2b59e91\") " pod="opendatahub/test-curl-7828-7a3e16"
Sep 17 00:32:31.315669 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:31.315638    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cqh9\" (UniqueName: \"kubernetes.io/projected/e42866cc-5509-4428-9434-cb71a2b59e91-kube-api-access-2cqh9\") pod \"test-curl-7828-7a3e16\" (UID: \"e42866cc-5509-4428-9434-cb71a2b59e91\") " pod="opendatahub/test-curl-7828-7a3e16"
Sep 17 00:32:31.449148 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:31.449108    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7828-7a3e16"
Sep 17 00:32:31.722039 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:31.722010    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7828-7a3e16"]
Sep 17 00:32:31.724304 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:32:31.724273    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode42866cc_5509_4428_9434_cb71a2b59e91.slice/crio-f374ba1c3a5555825975807536c649f00bad121d41fa468f4420cb58a68576fa WatchSource:0}: Error finding container f374ba1c3a5555825975807536c649f00bad121d41fa468f4420cb58a68576fa: Status 404 returned error can't find the container with id f374ba1c3a5555825975807536c649f00bad121d41fa468f4420cb58a68576fa
Sep 17 00:32:32.252001 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:32.251964    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7828-7a3e16" event={"ID":"e42866cc-5509-4428-9434-cb71a2b59e91","Type":"ContainerStarted","Data":"af1dd6509df7f756cf615da11a20e9b98b61cbfcf1a868e6363c75ad9d851e7c"}
Sep 17 00:32:32.252001 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:32.252007    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7828-7a3e16" event={"ID":"e42866cc-5509-4428-9434-cb71a2b59e91","Type":"ContainerStarted","Data":"f374ba1c3a5555825975807536c649f00bad121d41fa468f4420cb58a68576fa"}
Sep 17 00:32:32.266503 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:32.266446    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7828-7a3e16" podStartSLOduration=1.266427152 podStartE2EDuration="1.266427152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:32:32.264842328 +0000 UTC m=+1365.123335461" watchObservedRunningTime="2026-09-17 00:32:32.266427152 +0000 UTC m=+1365.124920278"
Sep 17 00:32:32.871137 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:32.871099    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7828-7a3e16"]
Sep 17 00:32:32.874813 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:32.874776    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7828-7a3e16"]
Sep 17 00:32:32.997646 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:32.997609    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7828-9bf77a"]
Sep 17 00:32:32.998043 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:32.998029    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e42866cc-5509-4428-9434-cb71a2b59e91" containerName="test-curl-7828-7a3e16"
Sep 17 00:32:32.998098 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:32.998044    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e42866cc-5509-4428-9434-cb71a2b59e91" containerName="test-curl-7828-7a3e16"
Sep 17 00:32:32.998134 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:32.998105    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e42866cc-5509-4428-9434-cb71a2b59e91" containerName="test-curl-7828-7a3e16"
Sep 17 00:32:33.001198 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:33.001177    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7828-9bf77a"
Sep 17 00:32:33.005281 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:33.005250    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7828-9bf77a"]
Sep 17 00:32:33.129640 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:33.129594    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jhvd\" (UniqueName: \"kubernetes.io/projected/2ecc7acc-3ccb-4f1b-9c3c-8f1822df0695-kube-api-access-4jhvd\") pod \"test-curl-7828-9bf77a\" (UID: \"2ecc7acc-3ccb-4f1b-9c3c-8f1822df0695\") " pod="opendatahub/test-curl-7828-9bf77a"
Sep 17 00:32:33.230609 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:33.230573    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jhvd\" (UniqueName: \"kubernetes.io/projected/2ecc7acc-3ccb-4f1b-9c3c-8f1822df0695-kube-api-access-4jhvd\") pod \"test-curl-7828-9bf77a\" (UID: \"2ecc7acc-3ccb-4f1b-9c3c-8f1822df0695\") " pod="opendatahub/test-curl-7828-9bf77a"
Sep 17 00:32:33.241885 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:33.241853    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jhvd\" (UniqueName: \"kubernetes.io/projected/2ecc7acc-3ccb-4f1b-9c3c-8f1822df0695-kube-api-access-4jhvd\") pod \"test-curl-7828-9bf77a\" (UID: \"2ecc7acc-3ccb-4f1b-9c3c-8f1822df0695\") " pod="opendatahub/test-curl-7828-9bf77a"
Sep 17 00:32:33.258654 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:32:33.258567    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7828-7a3e16" container="test-curl-7828-7a3e16"
Sep 17 00:32:33.313558 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:33.313519    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7828-9bf77a"
Sep 17 00:32:33.586487 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:33.586457    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7828-9bf77a"]
Sep 17 00:32:33.587721 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:32:33.587690    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ecc7acc_3ccb_4f1b_9c3c_8f1822df0695.slice/crio-9d8f81ce14e7c2c97974ad0a8038498fa06fc3cef424aabf336ffbf263e01555 WatchSource:0}: Error finding container 9d8f81ce14e7c2c97974ad0a8038498fa06fc3cef424aabf336ffbf263e01555: Status 404 returned error can't find the container with id 9d8f81ce14e7c2c97974ad0a8038498fa06fc3cef424aabf336ffbf263e01555
Sep 17 00:32:34.266393 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:34.266354    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7828-9bf77a" event={"ID":"2ecc7acc-3ccb-4f1b-9c3c-8f1822df0695","Type":"ContainerStarted","Data":"adbcf3a6f39712434c7684f282ada4ebba197eb4b50ad36e972fe106b9a9459d"}
Sep 17 00:32:34.266895 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:34.266407    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7828-9bf77a" event={"ID":"2ecc7acc-3ccb-4f1b-9c3c-8f1822df0695","Type":"ContainerStarted","Data":"9d8f81ce14e7c2c97974ad0a8038498fa06fc3cef424aabf336ffbf263e01555"}
Sep 17 00:32:34.266895 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:34.266479    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7828-7a3e16" podUID="e42866cc-5509-4428-9434-cb71a2b59e91" containerName="test-curl-7828-7a3e16" containerID="cri-o://af1dd6509df7f756cf615da11a20e9b98b61cbfcf1a868e6363c75ad9d851e7c" gracePeriod=2
Sep 17 00:32:34.279133 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:34.279073    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7828-9bf77a" podStartSLOduration=2.279053742 podStartE2EDuration="2.279053742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:32:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:32:34.278790582 +0000 UTC m=+1367.137283728" watchObservedRunningTime="2026-09-17 00:32:34.279053742 +0000 UTC m=+1367.137546878"
Sep 17 00:32:34.866233 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:34.866197    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7828-9bf77a"]
Sep 17 00:32:34.870728 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:34.870566    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7828-9bf77a"]
Sep 17 00:32:34.985776 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:34.985737    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7828-b85c49"]
Sep 17 00:32:34.986180 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:34.986161    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ecc7acc-3ccb-4f1b-9c3c-8f1822df0695" containerName="test-curl-7828-9bf77a"
Sep 17 00:32:34.986271 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:34.986182    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ecc7acc-3ccb-4f1b-9c3c-8f1822df0695" containerName="test-curl-7828-9bf77a"
Sep 17 00:32:34.986330 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:34.986302    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ecc7acc-3ccb-4f1b-9c3c-8f1822df0695" containerName="test-curl-7828-9bf77a"
Sep 17 00:32:34.989470 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:34.989448    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7828-b85c49"
Sep 17 00:32:34.993143 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:34.993114    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7828-b85c49"]
Sep 17 00:32:35.150869 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:35.150823    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zgsq\" (UniqueName: \"kubernetes.io/projected/768a9a3b-2809-49d1-8992-8b5a7a78c5bd-kube-api-access-9zgsq\") pod \"test-curl-7828-b85c49\" (UID: \"768a9a3b-2809-49d1-8992-8b5a7a78c5bd\") " pod="opendatahub/test-curl-7828-b85c49"
Sep 17 00:32:35.251792 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:35.251686    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zgsq\" (UniqueName: \"kubernetes.io/projected/768a9a3b-2809-49d1-8992-8b5a7a78c5bd-kube-api-access-9zgsq\") pod \"test-curl-7828-b85c49\" (UID: \"768a9a3b-2809-49d1-8992-8b5a7a78c5bd\") " pod="opendatahub/test-curl-7828-b85c49"
Sep 17 00:32:35.259360 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:35.259326    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zgsq\" (UniqueName: \"kubernetes.io/projected/768a9a3b-2809-49d1-8992-8b5a7a78c5bd-kube-api-access-9zgsq\") pod \"test-curl-7828-b85c49\" (UID: \"768a9a3b-2809-49d1-8992-8b5a7a78c5bd\") " pod="opendatahub/test-curl-7828-b85c49"
Sep 17 00:32:35.272309 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:32:35.272278    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7828-9bf77a" container="test-curl-7828-9bf77a"
Sep 17 00:32:35.300776 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:35.300736    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7828-b85c49"
Sep 17 00:32:35.572358 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:35.572323    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7828-b85c49"]
Sep 17 00:32:35.574666 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:32:35.574630    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod768a9a3b_2809_49d1_8992_8b5a7a78c5bd.slice/crio-62bdec68577835564fc3cab3af9b73fc7b573faff3b9430b764f24ae1a7b45d4 WatchSource:0}: Error finding container 62bdec68577835564fc3cab3af9b73fc7b573faff3b9430b764f24ae1a7b45d4: Status 404 returned error can't find the container with id 62bdec68577835564fc3cab3af9b73fc7b573faff3b9430b764f24ae1a7b45d4
Sep 17 00:32:36.279469 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:36.279400    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7828-b85c49" event={"ID":"768a9a3b-2809-49d1-8992-8b5a7a78c5bd","Type":"ContainerStarted","Data":"719e669cea766673a51e85dfd77c2be7684adaaaabe9e27752bb343276887790"}
Sep 17 00:32:36.279851 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:36.279485    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7828-b85c49" event={"ID":"768a9a3b-2809-49d1-8992-8b5a7a78c5bd","Type":"ContainerStarted","Data":"62bdec68577835564fc3cab3af9b73fc7b573faff3b9430b764f24ae1a7b45d4"}
Sep 17 00:32:36.279851 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:36.279543    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7828-9bf77a" podUID="2ecc7acc-3ccb-4f1b-9c3c-8f1822df0695" containerName="test-curl-7828-9bf77a" containerID="cri-o://adbcf3a6f39712434c7684f282ada4ebba197eb4b50ad36e972fe106b9a9459d" gracePeriod=2
Sep 17 00:32:36.295562 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:36.295505    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7828-b85c49" podStartSLOduration=2.295487704 podStartE2EDuration="2.295487704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:32:36.292505388 +0000 UTC m=+1369.150998523" watchObservedRunningTime="2026-09-17 00:32:36.295487704 +0000 UTC m=+1369.153980853"
Sep 17 00:32:36.461315 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:36.461283    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7828-7a3e16"
Sep 17 00:32:36.566625 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:36.566522    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cqh9\" (UniqueName: \"kubernetes.io/projected/e42866cc-5509-4428-9434-cb71a2b59e91-kube-api-access-2cqh9\") pod \"e42866cc-5509-4428-9434-cb71a2b59e91\" (UID: \"e42866cc-5509-4428-9434-cb71a2b59e91\") "
Sep 17 00:32:36.568715 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:36.568676    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42866cc-5509-4428-9434-cb71a2b59e91-kube-api-access-2cqh9" (OuterVolumeSpecName: "kube-api-access-2cqh9") pod "e42866cc-5509-4428-9434-cb71a2b59e91" (UID: "e42866cc-5509-4428-9434-cb71a2b59e91"). InnerVolumeSpecName "kube-api-access-2cqh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:32:36.668090 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:36.668065    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cqh9\" (UniqueName: \"kubernetes.io/projected/e42866cc-5509-4428-9434-cb71a2b59e91-kube-api-access-2cqh9\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:32:36.873515 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:36.873404    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7828-b85c49"]
Sep 17 00:32:36.876157 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:36.876126    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7828-b85c49"]
Sep 17 00:32:36.998809 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:36.998772    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7828-4c5d72"]
Sep 17 00:32:36.999177 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:36.999164    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="768a9a3b-2809-49d1-8992-8b5a7a78c5bd" containerName="test-curl-7828-b85c49"
Sep 17 00:32:36.999221 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:36.999179    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="768a9a3b-2809-49d1-8992-8b5a7a78c5bd" containerName="test-curl-7828-b85c49"
Sep 17 00:32:36.999257 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:36.999247    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="768a9a3b-2809-49d1-8992-8b5a7a78c5bd" containerName="test-curl-7828-b85c49"
Sep 17 00:32:37.002424 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:37.002398    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7828-4c5d72"
Sep 17 00:32:37.008442 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:37.008379    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7828-4c5d72"]
Sep 17 00:32:37.173519 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:37.173401    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-948cl\" (UniqueName: \"kubernetes.io/projected/bee05780-ccb5-4722-8a30-d4dddc9102e3-kube-api-access-948cl\") pod \"test-curl-7828-4c5d72\" (UID: \"bee05780-ccb5-4722-8a30-d4dddc9102e3\") " pod="opendatahub/test-curl-7828-4c5d72"
Sep 17 00:32:37.274443 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:37.274375    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-948cl\" (UniqueName: \"kubernetes.io/projected/bee05780-ccb5-4722-8a30-d4dddc9102e3-kube-api-access-948cl\") pod \"test-curl-7828-4c5d72\" (UID: \"bee05780-ccb5-4722-8a30-d4dddc9102e3\") " pod="opendatahub/test-curl-7828-4c5d72"
Sep 17 00:32:37.281947 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:37.281918    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-948cl\" (UniqueName: \"kubernetes.io/projected/bee05780-ccb5-4722-8a30-d4dddc9102e3-kube-api-access-948cl\") pod \"test-curl-7828-4c5d72\" (UID: \"bee05780-ccb5-4722-8a30-d4dddc9102e3\") " pod="opendatahub/test-curl-7828-4c5d72"
Sep 17 00:32:37.286379 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:37.286347    2577 generic.go:356] "Generic (PLEG): container finished" podID="e42866cc-5509-4428-9434-cb71a2b59e91" containerID="af1dd6509df7f756cf615da11a20e9b98b61cbfcf1a868e6363c75ad9d851e7c" exitCode=137
Sep 17 00:32:37.286514 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:37.286397    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7828-7a3e16"
Sep 17 00:32:37.286514 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:37.286453    2577 scope.go:117] "RemoveContainer" containerID="af1dd6509df7f756cf615da11a20e9b98b61cbfcf1a868e6363c75ad9d851e7c"
Sep 17 00:32:37.286763 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:32:37.286736    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7828-b85c49" container="test-curl-7828-b85c49"
Sep 17 00:32:37.296739 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:37.296716    2577 scope.go:117] "RemoveContainer" containerID="af1dd6509df7f756cf615da11a20e9b98b61cbfcf1a868e6363c75ad9d851e7c"
Sep 17 00:32:37.297073 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:32:37.297049    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af1dd6509df7f756cf615da11a20e9b98b61cbfcf1a868e6363c75ad9d851e7c\": container with ID starting with af1dd6509df7f756cf615da11a20e9b98b61cbfcf1a868e6363c75ad9d851e7c not found: ID does not exist" containerID="af1dd6509df7f756cf615da11a20e9b98b61cbfcf1a868e6363c75ad9d851e7c"
Sep 17 00:32:37.297122 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:37.297081    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af1dd6509df7f756cf615da11a20e9b98b61cbfcf1a868e6363c75ad9d851e7c"} err="failed to get container status \"af1dd6509df7f756cf615da11a20e9b98b61cbfcf1a868e6363c75ad9d851e7c\": rpc error: code = NotFound desc = could not find container \"af1dd6509df7f756cf615da11a20e9b98b61cbfcf1a868e6363c75ad9d851e7c\": container with ID starting with af1dd6509df7f756cf615da11a20e9b98b61cbfcf1a868e6363c75ad9d851e7c not found: ID does not exist"
Sep 17 00:32:37.298259 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:32:37.298236    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7828-7a3e16\" is forbidden: User \"system:node:ip-10-0-139-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-27.ec2.internal' and this object" podUID="e42866cc-5509-4428-9434-cb71a2b59e91" pod="opendatahub/test-curl-7828-7a3e16"
Sep 17 00:32:37.314487 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:37.314447    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7828-4c5d72"
Sep 17 00:32:37.588168 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:37.587905    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7828-4c5d72"]
Sep 17 00:32:37.590858 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:32:37.590829    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbee05780_ccb5_4722_8a30_d4dddc9102e3.slice/crio-9e29cf7f4b863dd3acc404014b566dc952d2afa1a4a2f291312e16f2cbd0e7b9 WatchSource:0}: Error finding container 9e29cf7f4b863dd3acc404014b566dc952d2afa1a4a2f291312e16f2cbd0e7b9: Status 404 returned error can't find the container with id 9e29cf7f4b863dd3acc404014b566dc952d2afa1a4a2f291312e16f2cbd0e7b9
Sep 17 00:32:37.609195 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:32:37.609162    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7828-7a3e16\" is forbidden: User \"system:node:ip-10-0-139-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-27.ec2.internal' and this object" podUID="e42866cc-5509-4428-9434-cb71a2b59e91" pod="opendatahub/test-curl-7828-7a3e16"
Sep 17 00:32:37.610797 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:37.610776    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e42866cc-5509-4428-9434-cb71a2b59e91" path="/var/lib/kubelet/pods/e42866cc-5509-4428-9434-cb71a2b59e91/volumes"
Sep 17 00:32:38.295451 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:38.295402    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7828-4c5d72" event={"ID":"bee05780-ccb5-4722-8a30-d4dddc9102e3","Type":"ContainerStarted","Data":"22ff32cfdd36f605dbf772f09bd454949a39eb60051e985e711c1defc6e9fb2f"}
Sep 17 00:32:38.295777 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:38.295463    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7828-4c5d72" event={"ID":"bee05780-ccb5-4722-8a30-d4dddc9102e3","Type":"ContainerStarted","Data":"9e29cf7f4b863dd3acc404014b566dc952d2afa1a4a2f291312e16f2cbd0e7b9"}
Sep 17 00:32:38.295777 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:38.295585    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7828-b85c49" podUID="768a9a3b-2809-49d1-8992-8b5a7a78c5bd" containerName="test-curl-7828-b85c49" containerID="cri-o://719e669cea766673a51e85dfd77c2be7684adaaaabe9e27752bb343276887790" gracePeriod=2
Sep 17 00:32:38.309071 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:38.309016    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7828-4c5d72" podStartSLOduration=2.309000622 podStartE2EDuration="2.309000622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:32:38.308372435 +0000 UTC m=+1371.166865571" watchObservedRunningTime="2026-09-17 00:32:38.309000622 +0000 UTC m=+1371.167493757"
Sep 17 00:32:38.471591 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:38.471563    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7828-9bf77a"
Sep 17 00:32:38.588129 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:38.588036    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jhvd\" (UniqueName: \"kubernetes.io/projected/2ecc7acc-3ccb-4f1b-9c3c-8f1822df0695-kube-api-access-4jhvd\") pod \"2ecc7acc-3ccb-4f1b-9c3c-8f1822df0695\" (UID: \"2ecc7acc-3ccb-4f1b-9c3c-8f1822df0695\") "
Sep 17 00:32:38.590163 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:38.590128    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ecc7acc-3ccb-4f1b-9c3c-8f1822df0695-kube-api-access-4jhvd" (OuterVolumeSpecName: "kube-api-access-4jhvd") pod "2ecc7acc-3ccb-4f1b-9c3c-8f1822df0695" (UID: "2ecc7acc-3ccb-4f1b-9c3c-8f1822df0695"). InnerVolumeSpecName "kube-api-access-4jhvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:32:38.689409 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:38.689381    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jhvd\" (UniqueName: \"kubernetes.io/projected/2ecc7acc-3ccb-4f1b-9c3c-8f1822df0695-kube-api-access-4jhvd\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:32:38.889242 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:38.889149    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7828-4c5d72"]
Sep 17 00:32:38.895328 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:38.895294    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7828-4c5d72"]
Sep 17 00:32:39.301772 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:39.301737    2577 generic.go:356] "Generic (PLEG): container finished" podID="2ecc7acc-3ccb-4f1b-9c3c-8f1822df0695" containerID="adbcf3a6f39712434c7684f282ada4ebba197eb4b50ad36e972fe106b9a9459d" exitCode=137
Sep 17 00:32:39.302244 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:39.301794    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7828-9bf77a"
Sep 17 00:32:39.302244 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:39.301838    2577 scope.go:117] "RemoveContainer" containerID="adbcf3a6f39712434c7684f282ada4ebba197eb4b50ad36e972fe106b9a9459d"
Sep 17 00:32:39.302244 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:32:39.302072    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7828-4c5d72" container="test-curl-7828-4c5d72"
Sep 17 00:32:39.311501 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:39.311484    2577 scope.go:117] "RemoveContainer" containerID="adbcf3a6f39712434c7684f282ada4ebba197eb4b50ad36e972fe106b9a9459d"
Sep 17 00:32:39.311808 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:32:39.311778    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adbcf3a6f39712434c7684f282ada4ebba197eb4b50ad36e972fe106b9a9459d\": container with ID starting with adbcf3a6f39712434c7684f282ada4ebba197eb4b50ad36e972fe106b9a9459d not found: ID does not exist" containerID="adbcf3a6f39712434c7684f282ada4ebba197eb4b50ad36e972fe106b9a9459d"
Sep 17 00:32:39.311899 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:39.311817    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adbcf3a6f39712434c7684f282ada4ebba197eb4b50ad36e972fe106b9a9459d"} err="failed to get container status \"adbcf3a6f39712434c7684f282ada4ebba197eb4b50ad36e972fe106b9a9459d\": rpc error: code = NotFound desc = could not find container \"adbcf3a6f39712434c7684f282ada4ebba197eb4b50ad36e972fe106b9a9459d\": container with ID starting with adbcf3a6f39712434c7684f282ada4ebba197eb4b50ad36e972fe106b9a9459d not found: ID does not exist"
Sep 17 00:32:39.312193 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:32:39.312171    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7828-9bf77a\" is forbidden: User \"system:node:ip-10-0-139-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-27.ec2.internal' and this object" podUID="2ecc7acc-3ccb-4f1b-9c3c-8f1822df0695" pod="opendatahub/test-curl-7828-9bf77a"
Sep 17 00:32:39.607808 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:39.607714    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ecc7acc-3ccb-4f1b-9c3c-8f1822df0695" path="/var/lib/kubelet/pods/2ecc7acc-3ccb-4f1b-9c3c-8f1822df0695/volumes"
Sep 17 00:32:40.309333 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:40.309276    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7828-4c5d72" podUID="bee05780-ccb5-4722-8a30-d4dddc9102e3" containerName="test-curl-7828-4c5d72" containerID="cri-o://22ff32cfdd36f605dbf772f09bd454949a39eb60051e985e711c1defc6e9fb2f" gracePeriod=2
Sep 17 00:32:40.500308 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:40.500277    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7828-b85c49"
Sep 17 00:32:40.608026 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:40.607925    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zgsq\" (UniqueName: \"kubernetes.io/projected/768a9a3b-2809-49d1-8992-8b5a7a78c5bd-kube-api-access-9zgsq\") pod \"768a9a3b-2809-49d1-8992-8b5a7a78c5bd\" (UID: \"768a9a3b-2809-49d1-8992-8b5a7a78c5bd\") "
Sep 17 00:32:40.610031 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:40.609997    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/768a9a3b-2809-49d1-8992-8b5a7a78c5bd-kube-api-access-9zgsq" (OuterVolumeSpecName: "kube-api-access-9zgsq") pod "768a9a3b-2809-49d1-8992-8b5a7a78c5bd" (UID: "768a9a3b-2809-49d1-8992-8b5a7a78c5bd"). InnerVolumeSpecName "kube-api-access-9zgsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:32:40.709624 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:40.709562    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zgsq\" (UniqueName: \"kubernetes.io/projected/768a9a3b-2809-49d1-8992-8b5a7a78c5bd-kube-api-access-9zgsq\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:32:41.317175 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:41.317136    2577 generic.go:356] "Generic (PLEG): container finished" podID="768a9a3b-2809-49d1-8992-8b5a7a78c5bd" containerID="719e669cea766673a51e85dfd77c2be7684adaaaabe9e27752bb343276887790" exitCode=137
Sep 17 00:32:41.317688 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:41.317183    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7828-b85c49"
Sep 17 00:32:41.317688 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:41.317191    2577 scope.go:117] "RemoveContainer" containerID="719e669cea766673a51e85dfd77c2be7684adaaaabe9e27752bb343276887790"
Sep 17 00:32:41.326983 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:41.326964    2577 scope.go:117] "RemoveContainer" containerID="719e669cea766673a51e85dfd77c2be7684adaaaabe9e27752bb343276887790"
Sep 17 00:32:41.327306 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:32:41.327286    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"719e669cea766673a51e85dfd77c2be7684adaaaabe9e27752bb343276887790\": container with ID starting with 719e669cea766673a51e85dfd77c2be7684adaaaabe9e27752bb343276887790 not found: ID does not exist" containerID="719e669cea766673a51e85dfd77c2be7684adaaaabe9e27752bb343276887790"
Sep 17 00:32:41.327354 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:41.327318    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"719e669cea766673a51e85dfd77c2be7684adaaaabe9e27752bb343276887790"} err="failed to get container status \"719e669cea766673a51e85dfd77c2be7684adaaaabe9e27752bb343276887790\": rpc error: code = NotFound desc = could not find container \"719e669cea766673a51e85dfd77c2be7684adaaaabe9e27752bb343276887790\": container with ID starting with 719e669cea766673a51e85dfd77c2be7684adaaaabe9e27752bb343276887790 not found: ID does not exist"
Sep 17 00:32:41.327889 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:32:41.327865    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7828-b85c49\" is forbidden: User \"system:node:ip-10-0-139-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-27.ec2.internal' and this object" podUID="768a9a3b-2809-49d1-8992-8b5a7a78c5bd" pod="opendatahub/test-curl-7828-b85c49"
Sep 17 00:32:41.599672 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:41.599588    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:32:41.608529 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:41.608502    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="768a9a3b-2809-49d1-8992-8b5a7a78c5bd" path="/var/lib/kubelet/pods/768a9a3b-2809-49d1-8992-8b5a7a78c5bd/volumes"
Sep 17 00:32:42.491818 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:42.491790    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7828-4c5d72"
Sep 17 00:32:42.525679 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:42.525639    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-948cl\" (UniqueName: \"kubernetes.io/projected/bee05780-ccb5-4722-8a30-d4dddc9102e3-kube-api-access-948cl\") pod \"bee05780-ccb5-4722-8a30-d4dddc9102e3\" (UID: \"bee05780-ccb5-4722-8a30-d4dddc9102e3\") "
Sep 17 00:32:42.527777 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:42.527747    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee05780-ccb5-4722-8a30-d4dddc9102e3-kube-api-access-948cl" (OuterVolumeSpecName: "kube-api-access-948cl") pod "bee05780-ccb5-4722-8a30-d4dddc9102e3" (UID: "bee05780-ccb5-4722-8a30-d4dddc9102e3"). InnerVolumeSpecName "kube-api-access-948cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:32:42.627063 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:42.626950    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-948cl\" (UniqueName: \"kubernetes.io/projected/bee05780-ccb5-4722-8a30-d4dddc9102e3-kube-api-access-948cl\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:32:43.333674 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:43.333637    2577 generic.go:356] "Generic (PLEG): container finished" podID="bee05780-ccb5-4722-8a30-d4dddc9102e3" containerID="22ff32cfdd36f605dbf772f09bd454949a39eb60051e985e711c1defc6e9fb2f" exitCode=137
Sep 17 00:32:43.333874 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:43.333700    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7828-4c5d72"
Sep 17 00:32:43.333874 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:43.333734    2577 scope.go:117] "RemoveContainer" containerID="22ff32cfdd36f605dbf772f09bd454949a39eb60051e985e711c1defc6e9fb2f"
Sep 17 00:32:43.343408 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:43.343379    2577 scope.go:117] "RemoveContainer" containerID="22ff32cfdd36f605dbf772f09bd454949a39eb60051e985e711c1defc6e9fb2f"
Sep 17 00:32:43.343812 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:32:43.343783    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22ff32cfdd36f605dbf772f09bd454949a39eb60051e985e711c1defc6e9fb2f\": container with ID starting with 22ff32cfdd36f605dbf772f09bd454949a39eb60051e985e711c1defc6e9fb2f not found: ID does not exist" containerID="22ff32cfdd36f605dbf772f09bd454949a39eb60051e985e711c1defc6e9fb2f"
Sep 17 00:32:43.343889 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:43.343824    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ff32cfdd36f605dbf772f09bd454949a39eb60051e985e711c1defc6e9fb2f"} err="failed to get container status \"22ff32cfdd36f605dbf772f09bd454949a39eb60051e985e711c1defc6e9fb2f\": rpc error: code = NotFound desc = could not find container \"22ff32cfdd36f605dbf772f09bd454949a39eb60051e985e711c1defc6e9fb2f\": container with ID starting with 22ff32cfdd36f605dbf772f09bd454949a39eb60051e985e711c1defc6e9fb2f not found: ID does not exist"
Sep 17 00:32:43.346596 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:32:43.346561    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7828-4c5d72\" is forbidden: User \"system:node:ip-10-0-139-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-27.ec2.internal' and this object" podUID="bee05780-ccb5-4722-8a30-d4dddc9102e3" pod="opendatahub/test-curl-7828-4c5d72"
Sep 17 00:32:43.608449 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:43.608347    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bee05780-ccb5-4722-8a30-d4dddc9102e3" path="/var/lib/kubelet/pods/bee05780-ccb5-4722-8a30-d4dddc9102e3/volumes"
Sep 17 00:32:46.831438 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:46.831377    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk"]
Sep 17 00:32:46.832000 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:46.831976    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bee05780-ccb5-4722-8a30-d4dddc9102e3" containerName="test-curl-7828-4c5d72"
Sep 17 00:32:46.832046 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:46.832006    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bee05780-ccb5-4722-8a30-d4dddc9102e3" containerName="test-curl-7828-4c5d72"
Sep 17 00:32:46.832125 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:46.832113    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bee05780-ccb5-4722-8a30-d4dddc9102e3" containerName="test-curl-7828-4c5d72"
Sep 17 00:32:46.837257 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:46.837228    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk"
Sep 17 00:32:46.839822 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:46.839785    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c5ae0fdd-serving-cert\""
Sep 17 00:32:46.843050 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:46.843025    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk"]
Sep 17 00:32:46.863660 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:46.863624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz5xk\" (UniqueName: \"kubernetes.io/projected/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf-kube-api-access-fz5xk\") pod \"maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk\" (UID: \"b1a0712f-1bc8-4147-a844-c6aa86ef9fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk"
Sep 17 00:32:46.863855 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:46.863742    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf-openshift-service-ca\") pod \"maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk\" (UID: \"b1a0712f-1bc8-4147-a844-c6aa86ef9fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk"
Sep 17 00:32:46.863855 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:46.863795    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf-maas-api-tls\") pod \"maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk\" (UID: \"b1a0712f-1bc8-4147-a844-c6aa86ef9fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk"
Sep 17 00:32:46.863855 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:46.863821    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf-metrics-certs\") pod \"maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk\" (UID: \"b1a0712f-1bc8-4147-a844-c6aa86ef9fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk"
Sep 17 00:32:46.964763 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:46.964718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz5xk\" (UniqueName: \"kubernetes.io/projected/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf-kube-api-access-fz5xk\") pod \"maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk\" (UID: \"b1a0712f-1bc8-4147-a844-c6aa86ef9fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk"
Sep 17 00:32:46.964979 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:46.964834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf-openshift-service-ca\") pod \"maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk\" (UID: \"b1a0712f-1bc8-4147-a844-c6aa86ef9fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk"
Sep 17 00:32:46.964979 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:46.964892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf-maas-api-tls\") pod \"maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk\" (UID: \"b1a0712f-1bc8-4147-a844-c6aa86ef9fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk"
Sep 17 00:32:46.964979 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:46.964922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf-metrics-certs\") pod \"maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk\" (UID: \"b1a0712f-1bc8-4147-a844-c6aa86ef9fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk"
Sep 17 00:32:46.965571 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:46.965542    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf-openshift-service-ca\") pod \"maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk\" (UID: \"b1a0712f-1bc8-4147-a844-c6aa86ef9fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk"
Sep 17 00:32:46.967567 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:46.967532    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf-maas-api-tls\") pod \"maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk\" (UID: \"b1a0712f-1bc8-4147-a844-c6aa86ef9fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk"
Sep 17 00:32:46.967666 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:46.967588    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf-metrics-certs\") pod \"maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk\" (UID: \"b1a0712f-1bc8-4147-a844-c6aa86ef9fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk"
Sep 17 00:32:46.972024 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:46.972001    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz5xk\" (UniqueName: \"kubernetes.io/projected/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf-kube-api-access-fz5xk\") pod \"maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk\" (UID: \"b1a0712f-1bc8-4147-a844-c6aa86ef9fcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk"
Sep 17 00:32:47.154030 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:47.153943    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk"
Sep 17 00:32:47.431190 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:47.431161    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk"]
Sep 17 00:32:47.433191 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:32:47.433168    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a0712f_1bc8_4147_a844_c6aa86ef9fcf.slice/crio-94c1962e2684931a35a18d147623800f45dc154a5b8dc99307bdd243f7daea8f WatchSource:0}: Error finding container 94c1962e2684931a35a18d147623800f45dc154a5b8dc99307bdd243f7daea8f: Status 404 returned error can't find the container with id 94c1962e2684931a35a18d147623800f45dc154a5b8dc99307bdd243f7daea8f
Sep 17 00:32:47.434904 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:47.434886    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 00:32:48.367990 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:48.367945    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk" event={"ID":"b1a0712f-1bc8-4147-a844-c6aa86ef9fcf","Type":"ContainerStarted","Data":"16163bc4646a787f0d6ca5df87fc357a6e28cdeb737d765746b29e45726ec7cf"}
Sep 17 00:32:48.368531 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:48.368014    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk" event={"ID":"b1a0712f-1bc8-4147-a844-c6aa86ef9fcf","Type":"ContainerStarted","Data":"94c1962e2684931a35a18d147623800f45dc154a5b8dc99307bdd243f7daea8f"}
Sep 17 00:32:48.368531 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:48.368039    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk"
Sep 17 00:32:48.385424 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:48.385356    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk" podStartSLOduration=1.9165585269999998 podStartE2EDuration="2.385340937s" totalImagesPullingTime="468.78241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:32:48.383742685 +0000 UTC m=+1381.242235825" watchObservedRunningTime="2026-09-17 00:32:48.385340937 +0000 UTC m=+1381.243834120"
Sep 17 00:32:51.784514 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:51.784466    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"]
Sep 17 00:32:54.379323 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:32:54.379289    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk"
Sep 17 00:33:04.094128 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.093957    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"]
Sep 17 00:33:04.097902 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.097878    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:04.101617 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.101593    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-090464\"/\"noten-model-090464-kserve-self-signed-certs\""
Sep 17 00:33:04.101756 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.101642    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-090464\"/\"openshift-service-ca.crt\""
Sep 17 00:33:04.101756 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.101647    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-090464\"/\"kube-root-ca.crt\""
Sep 17 00:33:04.101756 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.101654    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-090464\"/\"default-dockercfg-rf2ck\""
Sep 17 00:33:04.108213 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.108190    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"]
Sep 17 00:33:04.222165 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.222126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-tls-certs\") pod \"noten-model-090464-kserve-86697d4647-dp7td\" (UID: \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\") " pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:04.222333 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.222178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-model-cache\") pod \"noten-model-090464-kserve-86697d4647-dp7td\" (UID: \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\") " pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:04.222333 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.222208    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9k6p\" (UniqueName: \"kubernetes.io/projected/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-kube-api-access-g9k6p\") pod \"noten-model-090464-kserve-86697d4647-dp7td\" (UID: \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\") " pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:04.222333 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.222235    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-dshm\") pod \"noten-model-090464-kserve-86697d4647-dp7td\" (UID: \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\") " pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:04.222333 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.222252    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-home\") pod \"noten-model-090464-kserve-86697d4647-dp7td\" (UID: \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\") " pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:04.222333 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.222305    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-tmp-dir\") pod \"noten-model-090464-kserve-86697d4647-dp7td\" (UID: \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\") " pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:04.323081 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.323042    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-tls-certs\") pod \"noten-model-090464-kserve-86697d4647-dp7td\" (UID: \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\") " pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:04.323081 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.323085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-model-cache\") pod \"noten-model-090464-kserve-86697d4647-dp7td\" (UID: \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\") " pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:04.323348 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.323111    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9k6p\" (UniqueName: \"kubernetes.io/projected/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-kube-api-access-g9k6p\") pod \"noten-model-090464-kserve-86697d4647-dp7td\" (UID: \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\") " pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:04.323348 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.323148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-dshm\") pod \"noten-model-090464-kserve-86697d4647-dp7td\" (UID: \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\") " pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:04.323348 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.323171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-home\") pod \"noten-model-090464-kserve-86697d4647-dp7td\" (UID: \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\") " pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:04.323348 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.323206    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-tmp-dir\") pod \"noten-model-090464-kserve-86697d4647-dp7td\" (UID: \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\") " pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:04.323621 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.323526    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-model-cache\") pod \"noten-model-090464-kserve-86697d4647-dp7td\" (UID: \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\") " pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:04.323621 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.323539    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-home\") pod \"noten-model-090464-kserve-86697d4647-dp7td\" (UID: \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\") " pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:04.323621 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.323603    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-tmp-dir\") pod \"noten-model-090464-kserve-86697d4647-dp7td\" (UID: \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\") " pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:04.325491 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.325466    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-dshm\") pod \"noten-model-090464-kserve-86697d4647-dp7td\" (UID: \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\") " pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:04.325666 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.325649    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-tls-certs\") pod \"noten-model-090464-kserve-86697d4647-dp7td\" (UID: \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\") " pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:04.330785 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.330764    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9k6p\" (UniqueName: \"kubernetes.io/projected/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-kube-api-access-g9k6p\") pod \"noten-model-090464-kserve-86697d4647-dp7td\" (UID: \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\") " pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:04.409750 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.409677    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:04.681893 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:04.681866    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"]
Sep 17 00:33:04.683646 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:33:04.683617    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c8506aa_c189_4bd7_87f0_bc4136ee3cc5.slice/crio-e6da1d7b56f12c8b27429346bc3a0c9a13d1121f1e555d5632b67db92df535df WatchSource:0}: Error finding container e6da1d7b56f12c8b27429346bc3a0c9a13d1121f1e555d5632b67db92df535df: Status 404 returned error can't find the container with id e6da1d7b56f12c8b27429346bc3a0c9a13d1121f1e555d5632b67db92df535df
Sep 17 00:33:05.470632 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:05.470593    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td" event={"ID":"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5","Type":"ContainerStarted","Data":"62e66b85a61145328636da5bb68871b7422bdfebb84b2445f46b8753136110dc"}
Sep 17 00:33:05.471006 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:05.470644    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td" event={"ID":"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5","Type":"ContainerStarted","Data":"e6da1d7b56f12c8b27429346bc3a0c9a13d1121f1e555d5632b67db92df535df"}
Sep 17 00:33:05.487857 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:05.487808    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td" podStartSLOduration=1.487794598 podStartE2EDuration="1.487794598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:33:05.48527585 +0000 UTC m=+1398.343768986" watchObservedRunningTime="2026-09-17 00:33:05.487794598 +0000 UTC m=+1398.346287734"
Sep 17 00:33:06.804101 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:06.804062    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz" podUID="850c3459-5c5f-4b41-9aeb-905c098d4189" containerName="main" containerID="cri-o://341ccd84fd72b3ef9d924d77c0ec30d338e27f391c8e98001701cbf7932d74b7" gracePeriod=45
Sep 17 00:33:07.072924 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.072855    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:33:07.149624 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.149588    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbkm8\" (UniqueName: \"kubernetes.io/projected/850c3459-5c5f-4b41-9aeb-905c098d4189-kube-api-access-xbkm8\") pod \"850c3459-5c5f-4b41-9aeb-905c098d4189\" (UID: \"850c3459-5c5f-4b41-9aeb-905c098d4189\") "
Sep 17 00:33:07.149624 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.149631    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/850c3459-5c5f-4b41-9aeb-905c098d4189-tmp-dir\") pod \"850c3459-5c5f-4b41-9aeb-905c098d4189\" (UID: \"850c3459-5c5f-4b41-9aeb-905c098d4189\") "
Sep 17 00:33:07.149877 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.149751    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/850c3459-5c5f-4b41-9aeb-905c098d4189-home\") pod \"850c3459-5c5f-4b41-9aeb-905c098d4189\" (UID: \"850c3459-5c5f-4b41-9aeb-905c098d4189\") "
Sep 17 00:33:07.149877 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.149775    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/850c3459-5c5f-4b41-9aeb-905c098d4189-tls-certs\") pod \"850c3459-5c5f-4b41-9aeb-905c098d4189\" (UID: \"850c3459-5c5f-4b41-9aeb-905c098d4189\") "
Sep 17 00:33:07.149877 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.149816    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/850c3459-5c5f-4b41-9aeb-905c098d4189-dshm\") pod \"850c3459-5c5f-4b41-9aeb-905c098d4189\" (UID: \"850c3459-5c5f-4b41-9aeb-905c098d4189\") "
Sep 17 00:33:07.149877 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.149858    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/850c3459-5c5f-4b41-9aeb-905c098d4189-model-cache\") pod \"850c3459-5c5f-4b41-9aeb-905c098d4189\" (UID: \"850c3459-5c5f-4b41-9aeb-905c098d4189\") "
Sep 17 00:33:07.150115 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.149960    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850c3459-5c5f-4b41-9aeb-905c098d4189-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "850c3459-5c5f-4b41-9aeb-905c098d4189" (UID: "850c3459-5c5f-4b41-9aeb-905c098d4189"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:33:07.150115 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.150006    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850c3459-5c5f-4b41-9aeb-905c098d4189-home" (OuterVolumeSpecName: "home") pod "850c3459-5c5f-4b41-9aeb-905c098d4189" (UID: "850c3459-5c5f-4b41-9aeb-905c098d4189"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:33:07.150241 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.150179    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850c3459-5c5f-4b41-9aeb-905c098d4189-model-cache" (OuterVolumeSpecName: "model-cache") pod "850c3459-5c5f-4b41-9aeb-905c098d4189" (UID: "850c3459-5c5f-4b41-9aeb-905c098d4189"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:33:07.150315 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.150243    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/850c3459-5c5f-4b41-9aeb-905c098d4189-home\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:33:07.150315 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.150272    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/850c3459-5c5f-4b41-9aeb-905c098d4189-tmp-dir\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:33:07.152180 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.152148    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850c3459-5c5f-4b41-9aeb-905c098d4189-dshm" (OuterVolumeSpecName: "dshm") pod "850c3459-5c5f-4b41-9aeb-905c098d4189" (UID: "850c3459-5c5f-4b41-9aeb-905c098d4189"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:33:07.152367 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.152327    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850c3459-5c5f-4b41-9aeb-905c098d4189-kube-api-access-xbkm8" (OuterVolumeSpecName: "kube-api-access-xbkm8") pod "850c3459-5c5f-4b41-9aeb-905c098d4189" (UID: "850c3459-5c5f-4b41-9aeb-905c098d4189"). InnerVolumeSpecName "kube-api-access-xbkm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:33:07.152503 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.152394    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/850c3459-5c5f-4b41-9aeb-905c098d4189-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "850c3459-5c5f-4b41-9aeb-905c098d4189" (UID: "850c3459-5c5f-4b41-9aeb-905c098d4189"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:33:07.250937 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.250894    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbkm8\" (UniqueName: \"kubernetes.io/projected/850c3459-5c5f-4b41-9aeb-905c098d4189-kube-api-access-xbkm8\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:33:07.250937 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.250931    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/850c3459-5c5f-4b41-9aeb-905c098d4189-tls-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:33:07.251176 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.250948    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/850c3459-5c5f-4b41-9aeb-905c098d4189-dshm\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:33:07.251176 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.250960    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/850c3459-5c5f-4b41-9aeb-905c098d4189-model-cache\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:33:07.483887 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.483845    2577 generic.go:356] "Generic (PLEG): container finished" podID="850c3459-5c5f-4b41-9aeb-905c098d4189" containerID="341ccd84fd72b3ef9d924d77c0ec30d338e27f391c8e98001701cbf7932d74b7" exitCode=0
Sep 17 00:33:07.483887 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.483883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz" event={"ID":"850c3459-5c5f-4b41-9aeb-905c098d4189","Type":"ContainerDied","Data":"341ccd84fd72b3ef9d924d77c0ec30d338e27f391c8e98001701cbf7932d74b7"}
Sep 17 00:33:07.484126 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.483891    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"
Sep 17 00:33:07.484126 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.483924    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz" event={"ID":"850c3459-5c5f-4b41-9aeb-905c098d4189","Type":"ContainerDied","Data":"8fb91a99e3cd335df7717fa8f08009808a7215dd8f0c3e279204d8cb6bc43fb7"}
Sep 17 00:33:07.484126 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.483949    2577 scope.go:117] "RemoveContainer" containerID="341ccd84fd72b3ef9d924d77c0ec30d338e27f391c8e98001701cbf7932d74b7"
Sep 17 00:33:07.494195 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.494174    2577 scope.go:117] "RemoveContainer" containerID="341ccd84fd72b3ef9d924d77c0ec30d338e27f391c8e98001701cbf7932d74b7"
Sep 17 00:33:07.494502 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:33:07.494483    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"341ccd84fd72b3ef9d924d77c0ec30d338e27f391c8e98001701cbf7932d74b7\": container with ID starting with 341ccd84fd72b3ef9d924d77c0ec30d338e27f391c8e98001701cbf7932d74b7 not found: ID does not exist" containerID="341ccd84fd72b3ef9d924d77c0ec30d338e27f391c8e98001701cbf7932d74b7"
Sep 17 00:33:07.494573 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.494511    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341ccd84fd72b3ef9d924d77c0ec30d338e27f391c8e98001701cbf7932d74b7"} err="failed to get container status \"341ccd84fd72b3ef9d924d77c0ec30d338e27f391c8e98001701cbf7932d74b7\": rpc error: code = NotFound desc = could not find container \"341ccd84fd72b3ef9d924d77c0ec30d338e27f391c8e98001701cbf7932d74b7\": container with ID starting with 341ccd84fd72b3ef9d924d77c0ec30d338e27f391c8e98001701cbf7932d74b7 not found: ID does not exist"
Sep 17 00:33:07.505036 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.505015    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"]
Sep 17 00:33:07.508697 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.508674    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-76944c/test-model-76944c-kserve-6bf9bf4b67-6jgxz"]
Sep 17 00:33:07.607846 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:07.607815    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="850c3459-5c5f-4b41-9aeb-905c098d4189" path="/var/lib/kubelet/pods/850c3459-5c5f-4b41-9aeb-905c098d4189/volumes"
Sep 17 00:33:14.410697 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:14.410650    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:14.410697 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:14.410704    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:14.423751 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:14.423717    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:14.541195 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:14.541159    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:15.808306 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:15.808264    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:33:26.621510 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:26.621473    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-76944c-29826750-7gbnw"]
Sep 17 00:33:26.624979 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:26.624946    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-76944c-29826750-7gbnw"]
Sep 17 00:33:27.614205 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:27.614165    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0c5d985-3d7e-40cd-b53d-9b6c9441eeb1" path="/var/lib/kubelet/pods/d0c5d985-3d7e-40cd-b53d-9b6c9441eeb1/volumes"
Sep 17 00:33:30.001035 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.000988    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"]
Sep 17 00:33:30.001585 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.001563    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="850c3459-5c5f-4b41-9aeb-905c098d4189" containerName="main"
Sep 17 00:33:30.001665 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.001587    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="850c3459-5c5f-4b41-9aeb-905c098d4189" containerName="main"
Sep 17 00:33:30.001742 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.001730    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="850c3459-5c5f-4b41-9aeb-905c098d4189" containerName="main"
Sep 17 00:33:30.005016 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.004992    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:33:30.007390 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.007365    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2faea3\"/\"e2e-conflict-b-fe1bd81e-kserve-self-signed-certs\""
Sep 17 00:33:30.012323 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.012229    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"]
Sep 17 00:33:30.073760 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.073718    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-model-cache\") pod \"e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd\" (UID: \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\") " pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:33:30.073990 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.073768    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-home\") pod \"e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd\" (UID: \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\") " pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:33:30.073990 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.073889    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-dshm\") pod \"e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd\" (UID: \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\") " pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:33:30.073990 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.073929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk9v8\" (UniqueName: \"kubernetes.io/projected/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-kube-api-access-kk9v8\") pod \"e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd\" (UID: \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\") " pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:33:30.073990 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.073981    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-tls-certs\") pod \"e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd\" (UID: \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\") " pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:33:30.074184 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.074036    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-tmp-dir\") pod \"e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd\" (UID: \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\") " pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:33:30.175148 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.175106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-model-cache\") pod \"e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd\" (UID: \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\") " pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:33:30.175148 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.175155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-home\") pod \"e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd\" (UID: \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\") " pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:33:30.175484 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.175196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-dshm\") pod \"e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd\" (UID: \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\") " pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:33:30.175484 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.175216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk9v8\" (UniqueName: \"kubernetes.io/projected/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-kube-api-access-kk9v8\") pod \"e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd\" (UID: \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\") " pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:33:30.175484 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.175240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-tls-certs\") pod \"e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd\" (UID: \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\") " pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:33:30.175484 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.175283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-tmp-dir\") pod \"e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd\" (UID: \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\") " pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:33:30.175712 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.175663    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-model-cache\") pod \"e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd\" (UID: \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\") " pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:33:30.175712 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.175675    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-home\") pod \"e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd\" (UID: \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\") " pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:33:30.175783 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.175737    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-tmp-dir\") pod \"e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd\" (UID: \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\") " pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:33:30.177562 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.177540    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-dshm\") pod \"e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd\" (UID: \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\") " pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:33:30.177903 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.177884    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-tls-certs\") pod \"e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd\" (UID: \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\") " pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:33:30.183315 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.183288    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk9v8\" (UniqueName: \"kubernetes.io/projected/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-kube-api-access-kk9v8\") pod \"e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd\" (UID: \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\") " pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:33:30.315745 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.315647    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:33:30.596753 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.596718    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"]
Sep 17 00:33:30.598271 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:33:30.598247    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6a2a3a1_31b6_4a4b_a2ad_143701b41dc7.slice/crio-960244955cac24063442d180e05db465bedc52f53b7f904760a606978980986f WatchSource:0}: Error finding container 960244955cac24063442d180e05db465bedc52f53b7f904760a606978980986f: Status 404 returned error can't find the container with id 960244955cac24063442d180e05db465bedc52f53b7f904760a606978980986f
Sep 17 00:33:30.631730 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:30.631690    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd" event={"ID":"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7","Type":"ContainerStarted","Data":"960244955cac24063442d180e05db465bedc52f53b7f904760a606978980986f"}
Sep 17 00:33:31.646995 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:31.646953    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd" event={"ID":"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7","Type":"ContainerStarted","Data":"01fe0c81523ce519594a8bcef580db96c492cf6dc2cfd0733eac520660efce85"}
Sep 17 00:33:31.663094 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:31.663038    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd" podStartSLOduration=2.663022495 podStartE2EDuration="2.663022495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:33:31.661966209 +0000 UTC m=+1424.520459343" watchObservedRunningTime="2026-09-17 00:33:31.663022495 +0000 UTC m=+1424.521515656"
Sep 17 00:33:40.315849 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:40.315748    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:33:40.316276 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:40.316010    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:33:40.328709 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:40.328674    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:33:40.711326 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:40.711293    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:33:41.905912 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:41.905873    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:33:42.962053 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:42.962014    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"]
Sep 17 00:33:43.907547 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:43.907506    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb"]
Sep 17 00:33:43.907844 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:43.907811    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb" podUID="973c5c23-905c-4c92-bcdb-97b7600b449f" containerName="maas-api" containerID="cri-o://8eaede97f49e2de9c220252c43dbde52ac460987c70dd2d4b320d016a7584e5b" gracePeriod=30
Sep 17 00:33:44.017827 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:44.017746    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb" podUID="973c5c23-905c-4c92-bcdb-97b7600b449f" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.47:8443/health\": dial tcp 10.132.0.47:8443: connect: connection refused"
Sep 17 00:33:44.155154 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:44.155124    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb"
Sep 17 00:33:44.209749 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:44.209715    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4nn2\" (UniqueName: \"kubernetes.io/projected/973c5c23-905c-4c92-bcdb-97b7600b449f-kube-api-access-q4nn2\") pod \"973c5c23-905c-4c92-bcdb-97b7600b449f\" (UID: \"973c5c23-905c-4c92-bcdb-97b7600b449f\") "
Sep 17 00:33:44.209939 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:44.209781    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/973c5c23-905c-4c92-bcdb-97b7600b449f-maas-api-tls\") pod \"973c5c23-905c-4c92-bcdb-97b7600b449f\" (UID: \"973c5c23-905c-4c92-bcdb-97b7600b449f\") "
Sep 17 00:33:44.209939 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:44.209799    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/973c5c23-905c-4c92-bcdb-97b7600b449f-openshift-service-ca\") pod \"973c5c23-905c-4c92-bcdb-97b7600b449f\" (UID: \"973c5c23-905c-4c92-bcdb-97b7600b449f\") "
Sep 17 00:33:44.209939 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:44.209826    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/973c5c23-905c-4c92-bcdb-97b7600b449f-metrics-certs\") pod \"973c5c23-905c-4c92-bcdb-97b7600b449f\" (UID: \"973c5c23-905c-4c92-bcdb-97b7600b449f\") "
Sep 17 00:33:44.210216 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:44.210191    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/973c5c23-905c-4c92-bcdb-97b7600b449f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "973c5c23-905c-4c92-bcdb-97b7600b449f" (UID: "973c5c23-905c-4c92-bcdb-97b7600b449f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:33:44.211969 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:44.211937    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/973c5c23-905c-4c92-bcdb-97b7600b449f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "973c5c23-905c-4c92-bcdb-97b7600b449f" (UID: "973c5c23-905c-4c92-bcdb-97b7600b449f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:33:44.212088 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:44.211993    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/973c5c23-905c-4c92-bcdb-97b7600b449f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "973c5c23-905c-4c92-bcdb-97b7600b449f" (UID: "973c5c23-905c-4c92-bcdb-97b7600b449f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:33:44.212088 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:44.212043    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/973c5c23-905c-4c92-bcdb-97b7600b449f-kube-api-access-q4nn2" (OuterVolumeSpecName: "kube-api-access-q4nn2") pod "973c5c23-905c-4c92-bcdb-97b7600b449f" (UID: "973c5c23-905c-4c92-bcdb-97b7600b449f"). InnerVolumeSpecName "kube-api-access-q4nn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:33:44.311493 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:44.311449    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/973c5c23-905c-4c92-bcdb-97b7600b449f-maas-api-tls\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:33:44.311493 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:44.311489    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/973c5c23-905c-4c92-bcdb-97b7600b449f-openshift-service-ca\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:33:44.311493 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:44.311502    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/973c5c23-905c-4c92-bcdb-97b7600b449f-metrics-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:33:44.311744 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:44.311514    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4nn2\" (UniqueName: \"kubernetes.io/projected/973c5c23-905c-4c92-bcdb-97b7600b449f-kube-api-access-q4nn2\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:33:44.724384 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:44.724347    2577 generic.go:356] "Generic (PLEG): container finished" podID="973c5c23-905c-4c92-bcdb-97b7600b449f" containerID="8eaede97f49e2de9c220252c43dbde52ac460987c70dd2d4b320d016a7584e5b" exitCode=0
Sep 17 00:33:44.724580 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:44.724394    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb"
Sep 17 00:33:44.724580 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:44.724408    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb" event={"ID":"973c5c23-905c-4c92-bcdb-97b7600b449f","Type":"ContainerDied","Data":"8eaede97f49e2de9c220252c43dbde52ac460987c70dd2d4b320d016a7584e5b"}
Sep 17 00:33:44.724580 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:44.724486    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb" event={"ID":"973c5c23-905c-4c92-bcdb-97b7600b449f","Type":"ContainerDied","Data":"43a5cd982b9b6b584150331df87708f936072f8f058103f1098f3ef44d765a8f"}
Sep 17 00:33:44.724580 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:44.724517    2577 scope.go:117] "RemoveContainer" containerID="8eaede97f49e2de9c220252c43dbde52ac460987c70dd2d4b320d016a7584e5b"
Sep 17 00:33:44.734336 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:44.734316    2577 scope.go:117] "RemoveContainer" containerID="8eaede97f49e2de9c220252c43dbde52ac460987c70dd2d4b320d016a7584e5b"
Sep 17 00:33:44.734699 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:33:44.734674    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eaede97f49e2de9c220252c43dbde52ac460987c70dd2d4b320d016a7584e5b\": container with ID starting with 8eaede97f49e2de9c220252c43dbde52ac460987c70dd2d4b320d016a7584e5b not found: ID does not exist" containerID="8eaede97f49e2de9c220252c43dbde52ac460987c70dd2d4b320d016a7584e5b"
Sep 17 00:33:44.734766 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:44.734708    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eaede97f49e2de9c220252c43dbde52ac460987c70dd2d4b320d016a7584e5b"} err="failed to get container status \"8eaede97f49e2de9c220252c43dbde52ac460987c70dd2d4b320d016a7584e5b\": rpc error: code = NotFound desc = could not find container \"8eaede97f49e2de9c220252c43dbde52ac460987c70dd2d4b320d016a7584e5b\": container with ID starting with 8eaede97f49e2de9c220252c43dbde52ac460987c70dd2d4b320d016a7584e5b not found: ID does not exist"
Sep 17 00:33:44.746236 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:44.746202    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb"]
Sep 17 00:33:44.749842 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:44.749811    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a146d6-677f66845f-ndtbb"]
Sep 17 00:33:45.609007 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:45.608972    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="973c5c23-905c-4c92-bcdb-97b7600b449f" path="/var/lib/kubelet/pods/973c5c23-905c-4c92-bcdb-97b7600b449f/volumes"
Sep 17 00:33:57.982176 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:57.982109    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td" podUID="6c8506aa-c189-4bd7-87f0-bc4136ee3cc5" containerName="main" containerID="cri-o://62e66b85a61145328636da5bb68871b7422bdfebb84b2445f46b8753136110dc" gracePeriod=45
Sep 17 00:33:58.236688 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.236625    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:58.340570 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.340529    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9k6p\" (UniqueName: \"kubernetes.io/projected/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-kube-api-access-g9k6p\") pod \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\" (UID: \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\") "
Sep 17 00:33:58.340570 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.340577    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-tmp-dir\") pod \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\" (UID: \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\") "
Sep 17 00:33:58.340833 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.340608    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-dshm\") pod \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\" (UID: \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\") "
Sep 17 00:33:58.340833 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.340638    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-tls-certs\") pod \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\" (UID: \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\") "
Sep 17 00:33:58.340833 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.340667    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-home\") pod \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\" (UID: \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\") "
Sep 17 00:33:58.340833 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.340765    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-model-cache\") pod \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\" (UID: \"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5\") "
Sep 17 00:33:58.341036 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.340854    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c8506aa-c189-4bd7-87f0-bc4136ee3cc5" (UID: "6c8506aa-c189-4bd7-87f0-bc4136ee3cc5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:33:58.341036 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.340988    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-home" (OuterVolumeSpecName: "home") pod "6c8506aa-c189-4bd7-87f0-bc4136ee3cc5" (UID: "6c8506aa-c189-4bd7-87f0-bc4136ee3cc5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:33:58.341164 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.341072    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c8506aa-c189-4bd7-87f0-bc4136ee3cc5" (UID: "6c8506aa-c189-4bd7-87f0-bc4136ee3cc5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:33:58.341164 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.341154    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-model-cache\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:33:58.341273 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.341169    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-tmp-dir\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:33:58.341273 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.341177    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-home\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:33:58.342939 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.342913    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-dshm" (OuterVolumeSpecName: "dshm") pod "6c8506aa-c189-4bd7-87f0-bc4136ee3cc5" (UID: "6c8506aa-c189-4bd7-87f0-bc4136ee3cc5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:33:58.343063 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.342959    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c8506aa-c189-4bd7-87f0-bc4136ee3cc5" (UID: "6c8506aa-c189-4bd7-87f0-bc4136ee3cc5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:33:58.343063 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.342972    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-kube-api-access-g9k6p" (OuterVolumeSpecName: "kube-api-access-g9k6p") pod "6c8506aa-c189-4bd7-87f0-bc4136ee3cc5" (UID: "6c8506aa-c189-4bd7-87f0-bc4136ee3cc5"). InnerVolumeSpecName "kube-api-access-g9k6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:33:58.441811 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.441769    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9k6p\" (UniqueName: \"kubernetes.io/projected/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-kube-api-access-g9k6p\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:33:58.441811 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.441805    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-dshm\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:33:58.441811 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.441815    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5-tls-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:33:58.809538 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.809502    2577 generic.go:356] "Generic (PLEG): container finished" podID="6c8506aa-c189-4bd7-87f0-bc4136ee3cc5" containerID="62e66b85a61145328636da5bb68871b7422bdfebb84b2445f46b8753136110dc" exitCode=0
Sep 17 00:33:58.809732 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.809548    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"
Sep 17 00:33:58.809732 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.809587    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td" event={"ID":"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5","Type":"ContainerDied","Data":"62e66b85a61145328636da5bb68871b7422bdfebb84b2445f46b8753136110dc"}
Sep 17 00:33:58.809732 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.809627    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td" event={"ID":"6c8506aa-c189-4bd7-87f0-bc4136ee3cc5","Type":"ContainerDied","Data":"e6da1d7b56f12c8b27429346bc3a0c9a13d1121f1e555d5632b67db92df535df"}
Sep 17 00:33:58.809732 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.809646    2577 scope.go:117] "RemoveContainer" containerID="62e66b85a61145328636da5bb68871b7422bdfebb84b2445f46b8753136110dc"
Sep 17 00:33:58.819280 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.819259    2577 scope.go:117] "RemoveContainer" containerID="62e66b85a61145328636da5bb68871b7422bdfebb84b2445f46b8753136110dc"
Sep 17 00:33:58.819616 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:33:58.819594    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62e66b85a61145328636da5bb68871b7422bdfebb84b2445f46b8753136110dc\": container with ID starting with 62e66b85a61145328636da5bb68871b7422bdfebb84b2445f46b8753136110dc not found: ID does not exist" containerID="62e66b85a61145328636da5bb68871b7422bdfebb84b2445f46b8753136110dc"
Sep 17 00:33:58.819727 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.819624    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e66b85a61145328636da5bb68871b7422bdfebb84b2445f46b8753136110dc"} err="failed to get container status \"62e66b85a61145328636da5bb68871b7422bdfebb84b2445f46b8753136110dc\": rpc error: code = NotFound desc = could not find container \"62e66b85a61145328636da5bb68871b7422bdfebb84b2445f46b8753136110dc\": container with ID starting with 62e66b85a61145328636da5bb68871b7422bdfebb84b2445f46b8753136110dc not found: ID does not exist"
Sep 17 00:33:58.830403 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.830368    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"]
Sep 17 00:33:58.835810 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:58.835770    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-090464/noten-model-090464-kserve-86697d4647-dp7td"]
Sep 17 00:33:59.608429 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:33:59.608382    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c8506aa-c189-4bd7-87f0-bc4136ee3cc5" path="/var/lib/kubelet/pods/6c8506aa-c189-4bd7-87f0-bc4136ee3cc5/volumes"
Sep 17 00:34:03.892461 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:03.892426    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"]
Sep 17 00:34:03.892838 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:03.892809    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="973c5c23-905c-4c92-bcdb-97b7600b449f" containerName="maas-api"
Sep 17 00:34:03.892838 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:03.892824    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="973c5c23-905c-4c92-bcdb-97b7600b449f" containerName="maas-api"
Sep 17 00:34:03.892838 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:03.892838    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c8506aa-c189-4bd7-87f0-bc4136ee3cc5" containerName="main"
Sep 17 00:34:03.892944 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:03.892843    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c8506aa-c189-4bd7-87f0-bc4136ee3cc5" containerName="main"
Sep 17 00:34:03.892944 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:03.892921    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c8506aa-c189-4bd7-87f0-bc4136ee3cc5" containerName="main"
Sep 17 00:34:03.892944 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:03.892930    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="973c5c23-905c-4c92-bcdb-97b7600b449f" containerName="maas-api"
Sep 17 00:34:03.897410 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:03.897385    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"
Sep 17 00:34:03.899925 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:03.899895    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-47a2db-serving-cert\""
Sep 17 00:34:03.907716 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:03.907676    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"]
Sep 17 00:34:04.001013 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:04.000969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w2pw\" (UniqueName: \"kubernetes.io/projected/cae07c52-98a2-4244-9d3e-05dd95f8c86d-kube-api-access-4w2pw\") pod \"maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v\" (UID: \"cae07c52-98a2-4244-9d3e-05dd95f8c86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"
Sep 17 00:34:04.001196 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:04.001035    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae07c52-98a2-4244-9d3e-05dd95f8c86d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v\" (UID: \"cae07c52-98a2-4244-9d3e-05dd95f8c86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"
Sep 17 00:34:04.001196 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:04.001116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae07c52-98a2-4244-9d3e-05dd95f8c86d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v\" (UID: \"cae07c52-98a2-4244-9d3e-05dd95f8c86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"
Sep 17 00:34:04.001196 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:04.001140    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae07c52-98a2-4244-9d3e-05dd95f8c86d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v\" (UID: \"cae07c52-98a2-4244-9d3e-05dd95f8c86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"
Sep 17 00:34:04.102531 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:04.102474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae07c52-98a2-4244-9d3e-05dd95f8c86d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v\" (UID: \"cae07c52-98a2-4244-9d3e-05dd95f8c86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"
Sep 17 00:34:04.102728 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:04.102603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae07c52-98a2-4244-9d3e-05dd95f8c86d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v\" (UID: \"cae07c52-98a2-4244-9d3e-05dd95f8c86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"
Sep 17 00:34:04.102728 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:04.102639    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae07c52-98a2-4244-9d3e-05dd95f8c86d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v\" (UID: \"cae07c52-98a2-4244-9d3e-05dd95f8c86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"
Sep 17 00:34:04.102728 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:04.102690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w2pw\" (UniqueName: \"kubernetes.io/projected/cae07c52-98a2-4244-9d3e-05dd95f8c86d-kube-api-access-4w2pw\") pod \"maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v\" (UID: \"cae07c52-98a2-4244-9d3e-05dd95f8c86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"
Sep 17 00:34:04.102842 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:34:04.102756    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-serving-cert: secret "maas-api-e2e-shared-a-w3-47a2db-serving-cert" not found
Sep 17 00:34:04.102877 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:34:04.102857    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cae07c52-98a2-4244-9d3e-05dd95f8c86d-maas-api-tls podName:cae07c52-98a2-4244-9d3e-05dd95f8c86d nodeName:}" failed. No retries permitted until 2026-09-17 00:34:04.602838971 +0000 UTC m=+1457.461332085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cae07c52-98a2-4244-9d3e-05dd95f8c86d-maas-api-tls") pod "maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v" (UID: "cae07c52-98a2-4244-9d3e-05dd95f8c86d") : secret "maas-api-e2e-shared-a-w3-47a2db-serving-cert" not found
Sep 17 00:34:04.103289 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:04.103252    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae07c52-98a2-4244-9d3e-05dd95f8c86d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v\" (UID: \"cae07c52-98a2-4244-9d3e-05dd95f8c86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"
Sep 17 00:34:04.105129 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:04.105105    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae07c52-98a2-4244-9d3e-05dd95f8c86d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v\" (UID: \"cae07c52-98a2-4244-9d3e-05dd95f8c86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"
Sep 17 00:34:04.113579 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:04.113552    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w2pw\" (UniqueName: \"kubernetes.io/projected/cae07c52-98a2-4244-9d3e-05dd95f8c86d-kube-api-access-4w2pw\") pod \"maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v\" (UID: \"cae07c52-98a2-4244-9d3e-05dd95f8c86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"
Sep 17 00:34:04.608034 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:04.607993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae07c52-98a2-4244-9d3e-05dd95f8c86d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v\" (UID: \"cae07c52-98a2-4244-9d3e-05dd95f8c86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"
Sep 17 00:34:04.610397 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:04.610375    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae07c52-98a2-4244-9d3e-05dd95f8c86d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v\" (UID: \"cae07c52-98a2-4244-9d3e-05dd95f8c86d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"
Sep 17 00:34:04.808988 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:04.808942    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"
Sep 17 00:34:05.085300 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:05.085268    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"]
Sep 17 00:34:05.854123 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:05.854013    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v" event={"ID":"cae07c52-98a2-4244-9d3e-05dd95f8c86d","Type":"ContainerStarted","Data":"74aa15f1e0985a078e34d09465c202fedea086083c5177c7217a08b862308d80"}
Sep 17 00:34:05.854123 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:05.854084    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v" event={"ID":"cae07c52-98a2-4244-9d3e-05dd95f8c86d","Type":"ContainerStarted","Data":"7ed827e8f08085c1701b1f7b85f0028caf2c3159b522781305c0de4b5edf1768"}
Sep 17 00:34:05.854345 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:05.854158    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"
Sep 17 00:34:05.871731 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:05.871683    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v" podStartSLOduration=2.512259424 podStartE2EDuration="2.871665723s" totalImagesPullingTime="359.406299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:34:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:05.86960676 +0000 UTC m=+1458.728099895" watchObservedRunningTime="2026-09-17 00:34:05.871665723 +0000 UTC m=+1458.730158858"
Sep 17 00:34:11.866545 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:11.866514    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"
Sep 17 00:34:17.804511 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:17.804476    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:34:24.694807 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.694767    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"]
Sep 17 00:34:24.699077 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.699050    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:34:24.702686 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.702648    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c5ae0fdd\"/\"default-dockercfg-2bn4g\""
Sep 17 00:34:24.702838 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.702684    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c5ae0fdd\"/\"openshift-service-ca.crt\""
Sep 17 00:34:24.702838 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.702648    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c5ae0fdd\"/\"kube-root-ca.crt\""
Sep 17 00:34:24.702838 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.702648    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c5ae0fdd\"/\"e2e-lifecycle-model-c5ae0fdd-kserve-self-signed-certs\""
Sep 17 00:34:24.712801 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.712618    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"]
Sep 17 00:34:24.773944 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.773901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29ea83c6-01e8-4478-8ce0-ec50b6749561-model-cache\") pod \"e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49\" (UID: \"29ea83c6-01e8-4478-8ce0-ec50b6749561\") " pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:34:24.773944 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.773950    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29ea83c6-01e8-4478-8ce0-ec50b6749561-tmp-dir\") pod \"e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49\" (UID: \"29ea83c6-01e8-4478-8ce0-ec50b6749561\") " pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:34:24.774165 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.773971    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29ea83c6-01e8-4478-8ce0-ec50b6749561-home\") pod \"e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49\" (UID: \"29ea83c6-01e8-4478-8ce0-ec50b6749561\") " pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:34:24.774165 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.774012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzrkd\" (UniqueName: \"kubernetes.io/projected/29ea83c6-01e8-4478-8ce0-ec50b6749561-kube-api-access-bzrkd\") pod \"e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49\" (UID: \"29ea83c6-01e8-4478-8ce0-ec50b6749561\") " pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:34:24.774165 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.774090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29ea83c6-01e8-4478-8ce0-ec50b6749561-dshm\") pod \"e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49\" (UID: \"29ea83c6-01e8-4478-8ce0-ec50b6749561\") " pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:34:24.774165 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.774118    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29ea83c6-01e8-4478-8ce0-ec50b6749561-tls-certs\") pod \"e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49\" (UID: \"29ea83c6-01e8-4478-8ce0-ec50b6749561\") " pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:34:24.875170 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.875124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29ea83c6-01e8-4478-8ce0-ec50b6749561-model-cache\") pod \"e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49\" (UID: \"29ea83c6-01e8-4478-8ce0-ec50b6749561\") " pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:34:24.875170 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.875175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29ea83c6-01e8-4478-8ce0-ec50b6749561-tmp-dir\") pod \"e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49\" (UID: \"29ea83c6-01e8-4478-8ce0-ec50b6749561\") " pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:34:24.875447 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.875199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29ea83c6-01e8-4478-8ce0-ec50b6749561-home\") pod \"e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49\" (UID: \"29ea83c6-01e8-4478-8ce0-ec50b6749561\") " pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:34:24.875447 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.875217    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzrkd\" (UniqueName: \"kubernetes.io/projected/29ea83c6-01e8-4478-8ce0-ec50b6749561-kube-api-access-bzrkd\") pod \"e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49\" (UID: \"29ea83c6-01e8-4478-8ce0-ec50b6749561\") " pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:34:24.875447 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.875279    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29ea83c6-01e8-4478-8ce0-ec50b6749561-dshm\") pod \"e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49\" (UID: \"29ea83c6-01e8-4478-8ce0-ec50b6749561\") " pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:34:24.875447 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.875408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29ea83c6-01e8-4478-8ce0-ec50b6749561-tls-certs\") pod \"e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49\" (UID: \"29ea83c6-01e8-4478-8ce0-ec50b6749561\") " pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:34:24.875675 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.875573    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29ea83c6-01e8-4478-8ce0-ec50b6749561-model-cache\") pod \"e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49\" (UID: \"29ea83c6-01e8-4478-8ce0-ec50b6749561\") " pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:34:24.875675 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.875608    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29ea83c6-01e8-4478-8ce0-ec50b6749561-home\") pod \"e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49\" (UID: \"29ea83c6-01e8-4478-8ce0-ec50b6749561\") " pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:34:24.875774 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.875677    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29ea83c6-01e8-4478-8ce0-ec50b6749561-tmp-dir\") pod \"e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49\" (UID: \"29ea83c6-01e8-4478-8ce0-ec50b6749561\") " pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:34:24.877707 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.877684    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29ea83c6-01e8-4478-8ce0-ec50b6749561-dshm\") pod \"e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49\" (UID: \"29ea83c6-01e8-4478-8ce0-ec50b6749561\") " pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:34:24.878086 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.878060    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29ea83c6-01e8-4478-8ce0-ec50b6749561-tls-certs\") pod \"e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49\" (UID: \"29ea83c6-01e8-4478-8ce0-ec50b6749561\") " pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:34:24.882743 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:24.882721    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzrkd\" (UniqueName: \"kubernetes.io/projected/29ea83c6-01e8-4478-8ce0-ec50b6749561-kube-api-access-bzrkd\") pod \"e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49\" (UID: \"29ea83c6-01e8-4478-8ce0-ec50b6749561\") " pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:34:25.018482 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:25.018341    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:34:25.293489 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:25.293455    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"]
Sep 17 00:34:25.294681 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:34:25.294656    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29ea83c6_01e8_4478_8ce0_ec50b6749561.slice/crio-8f04c546e9a0aca2b6ece385829f2bf824f0800567989ae504485a2b5fab14d8 WatchSource:0}: Error finding container 8f04c546e9a0aca2b6ece385829f2bf824f0800567989ae504485a2b5fab14d8: Status 404 returned error can't find the container with id 8f04c546e9a0aca2b6ece385829f2bf824f0800567989ae504485a2b5fab14d8
Sep 17 00:34:25.970687 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:25.970648    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49" event={"ID":"29ea83c6-01e8-4478-8ce0-ec50b6749561","Type":"ContainerStarted","Data":"e4c0291dee39d17f4752455adb33c1f4cc20090a2284d6b57cc7575b8b1d88ee"}
Sep 17 00:34:25.971172 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:25.970697    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49" event={"ID":"29ea83c6-01e8-4478-8ce0-ec50b6749561","Type":"ContainerStarted","Data":"8f04c546e9a0aca2b6ece385829f2bf824f0800567989ae504485a2b5fab14d8"}
Sep 17 00:34:25.986558 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:25.986502    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49" podStartSLOduration=1.9864842440000001 podStartE2EDuration="1.986484244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:25.985792868 +0000 UTC m=+1478.844286003" watchObservedRunningTime="2026-09-17 00:34:25.986484244 +0000 UTC m=+1478.844977379"
Sep 17 00:34:35.018963 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:35.018918    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:34:35.019459 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:35.018979    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:34:35.031918 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:35.031887    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:34:35.043315 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:35.043285    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:34:47.666246 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:47.666218    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/ovn-acl-logging/0.log"
Sep 17 00:34:47.672399 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:47.672376    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/ovn-acl-logging/0.log"
Sep 17 00:34:53.010792 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:53.010753    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:34:53.913818 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:34:53.913781    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"]
Sep 17 00:35:04.005694 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:04.005648    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"]
Sep 17 00:35:04.009364 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:04.009343    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:35:04.012209 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:04.012184    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2faea3\"/\"distinct-w4-2faea3-kserve-self-signed-certs\""
Sep 17 00:35:04.018946 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:04.018917    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"]
Sep 17 00:35:04.054601 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:04.054569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63be31f1-7be5-4cc6-a614-271adaa92912-tmp-dir\") pod \"distinct-w4-2faea3-kserve-64678c55cc-dlgn8\" (UID: \"63be31f1-7be5-4cc6-a614-271adaa92912\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:35:04.054801 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:04.054612    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63be31f1-7be5-4cc6-a614-271adaa92912-dshm\") pod \"distinct-w4-2faea3-kserve-64678c55cc-dlgn8\" (UID: \"63be31f1-7be5-4cc6-a614-271adaa92912\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:35:04.054801 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:04.054634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mfs9\" (UniqueName: \"kubernetes.io/projected/63be31f1-7be5-4cc6-a614-271adaa92912-kube-api-access-6mfs9\") pod \"distinct-w4-2faea3-kserve-64678c55cc-dlgn8\" (UID: \"63be31f1-7be5-4cc6-a614-271adaa92912\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:35:04.054801 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:04.054709    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63be31f1-7be5-4cc6-a614-271adaa92912-model-cache\") pod \"distinct-w4-2faea3-kserve-64678c55cc-dlgn8\" (UID: \"63be31f1-7be5-4cc6-a614-271adaa92912\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:35:04.054933 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:04.054803    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63be31f1-7be5-4cc6-a614-271adaa92912-tls-certs\") pod \"distinct-w4-2faea3-kserve-64678c55cc-dlgn8\" (UID: \"63be31f1-7be5-4cc6-a614-271adaa92912\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:35:04.054933 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:04.054852    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63be31f1-7be5-4cc6-a614-271adaa92912-home\") pod \"distinct-w4-2faea3-kserve-64678c55cc-dlgn8\" (UID: \"63be31f1-7be5-4cc6-a614-271adaa92912\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:35:04.155783 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:04.155747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63be31f1-7be5-4cc6-a614-271adaa92912-dshm\") pod \"distinct-w4-2faea3-kserve-64678c55cc-dlgn8\" (UID: \"63be31f1-7be5-4cc6-a614-271adaa92912\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:35:04.155967 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:04.155796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mfs9\" (UniqueName: \"kubernetes.io/projected/63be31f1-7be5-4cc6-a614-271adaa92912-kube-api-access-6mfs9\") pod \"distinct-w4-2faea3-kserve-64678c55cc-dlgn8\" (UID: \"63be31f1-7be5-4cc6-a614-271adaa92912\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:35:04.155967 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:04.155903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63be31f1-7be5-4cc6-a614-271adaa92912-model-cache\") pod \"distinct-w4-2faea3-kserve-64678c55cc-dlgn8\" (UID: \"63be31f1-7be5-4cc6-a614-271adaa92912\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:35:04.156088 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:04.155971    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63be31f1-7be5-4cc6-a614-271adaa92912-tls-certs\") pod \"distinct-w4-2faea3-kserve-64678c55cc-dlgn8\" (UID: \"63be31f1-7be5-4cc6-a614-271adaa92912\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:35:04.156088 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:04.156026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63be31f1-7be5-4cc6-a614-271adaa92912-home\") pod \"distinct-w4-2faea3-kserve-64678c55cc-dlgn8\" (UID: \"63be31f1-7be5-4cc6-a614-271adaa92912\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:35:04.156195 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:04.156138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63be31f1-7be5-4cc6-a614-271adaa92912-tmp-dir\") pod \"distinct-w4-2faea3-kserve-64678c55cc-dlgn8\" (UID: \"63be31f1-7be5-4cc6-a614-271adaa92912\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:35:04.156552 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:04.156528    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63be31f1-7be5-4cc6-a614-271adaa92912-tmp-dir\") pod \"distinct-w4-2faea3-kserve-64678c55cc-dlgn8\" (UID: \"63be31f1-7be5-4cc6-a614-271adaa92912\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:35:04.156660 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:04.156526    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63be31f1-7be5-4cc6-a614-271adaa92912-home\") pod \"distinct-w4-2faea3-kserve-64678c55cc-dlgn8\" (UID: \"63be31f1-7be5-4cc6-a614-271adaa92912\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:35:04.156804 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:04.156786    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63be31f1-7be5-4cc6-a614-271adaa92912-model-cache\") pod \"distinct-w4-2faea3-kserve-64678c55cc-dlgn8\" (UID: \"63be31f1-7be5-4cc6-a614-271adaa92912\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:35:04.158174 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:04.158149    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63be31f1-7be5-4cc6-a614-271adaa92912-dshm\") pod \"distinct-w4-2faea3-kserve-64678c55cc-dlgn8\" (UID: \"63be31f1-7be5-4cc6-a614-271adaa92912\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:35:04.158945 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:04.158923    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63be31f1-7be5-4cc6-a614-271adaa92912-tls-certs\") pod \"distinct-w4-2faea3-kserve-64678c55cc-dlgn8\" (UID: \"63be31f1-7be5-4cc6-a614-271adaa92912\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:35:04.163248 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:04.163226    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mfs9\" (UniqueName: \"kubernetes.io/projected/63be31f1-7be5-4cc6-a614-271adaa92912-kube-api-access-6mfs9\") pod \"distinct-w4-2faea3-kserve-64678c55cc-dlgn8\" (UID: \"63be31f1-7be5-4cc6-a614-271adaa92912\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:35:04.319821 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:04.319720    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:35:04.594921 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:04.594891    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"]
Sep 17 00:35:04.597030 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:35:04.597002    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63be31f1_7be5_4cc6_a614_271adaa92912.slice/crio-08b668680f370a143cbe4dcf91c8281dc19ecc95e26656294aed8594e22c2ea6 WatchSource:0}: Error finding container 08b668680f370a143cbe4dcf91c8281dc19ecc95e26656294aed8594e22c2ea6: Status 404 returned error can't find the container with id 08b668680f370a143cbe4dcf91c8281dc19ecc95e26656294aed8594e22c2ea6
Sep 17 00:35:05.219309 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:05.219272    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8" event={"ID":"63be31f1-7be5-4cc6-a614-271adaa92912","Type":"ContainerStarted","Data":"3349f89d6ee5a70aba77cf304176b23e308f5462b2306735ec317d8ce18b10d6"}
Sep 17 00:35:05.219309 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:05.219314    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8" event={"ID":"63be31f1-7be5-4cc6-a614-271adaa92912","Type":"ContainerStarted","Data":"08b668680f370a143cbe4dcf91c8281dc19ecc95e26656294aed8594e22c2ea6"}
Sep 17 00:35:05.235679 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:05.235624    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8" podStartSLOduration=2.235609321 podStartE2EDuration="2.235609321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:35:05.23431752 +0000 UTC m=+1518.092810668" watchObservedRunningTime="2026-09-17 00:35:05.235609321 +0000 UTC m=+1518.094102476"
Sep 17 00:35:08.933995 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:08.933949    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd" podUID="f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7" containerName="main" containerID="cri-o://01fe0c81523ce519594a8bcef580db96c492cf6dc2cfd0733eac520660efce85" gracePeriod=45
Sep 17 00:35:09.184852 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.184774    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:35:09.204324 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.204289    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-tls-certs\") pod \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\" (UID: \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\") "
Sep 17 00:35:09.204527 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.204332    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kk9v8\" (UniqueName: \"kubernetes.io/projected/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-kube-api-access-kk9v8\") pod \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\" (UID: \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\") "
Sep 17 00:35:09.204527 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.204362    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-tmp-dir\") pod \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\" (UID: \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\") "
Sep 17 00:35:09.204527 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.204389    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-home\") pod \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\" (UID: \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\") "
Sep 17 00:35:09.204527 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.204437    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-dshm\") pod \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\" (UID: \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\") "
Sep 17 00:35:09.204767 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.204538    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-model-cache\") pod \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\" (UID: \"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7\") "
Sep 17 00:35:09.204767 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.204710    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7" (UID: "f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:35:09.204767 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.204750    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-home" (OuterVolumeSpecName: "home") pod "f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7" (UID: "f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:35:09.204959 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.204929    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7" (UID: "f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:35:09.205027 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.205004    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-tmp-dir\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:35:09.205027 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.205024    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-home\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:35:09.206943 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.206916    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-dshm" (OuterVolumeSpecName: "dshm") pod "f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7" (UID: "f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:35:09.207052 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.206949    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-kube-api-access-kk9v8" (OuterVolumeSpecName: "kube-api-access-kk9v8") pod "f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7" (UID: "f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7"). InnerVolumeSpecName "kube-api-access-kk9v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:35:09.207052 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.207015    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7" (UID: "f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:35:09.247301 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.247085    2577 generic.go:356] "Generic (PLEG): container finished" podID="f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7" containerID="01fe0c81523ce519594a8bcef580db96c492cf6dc2cfd0733eac520660efce85" exitCode=0
Sep 17 00:35:09.247301 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.247217    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd" event={"ID":"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7","Type":"ContainerDied","Data":"01fe0c81523ce519594a8bcef580db96c492cf6dc2cfd0733eac520660efce85"}
Sep 17 00:35:09.247301 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.247253    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd" event={"ID":"f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7","Type":"ContainerDied","Data":"960244955cac24063442d180e05db465bedc52f53b7f904760a606978980986f"}
Sep 17 00:35:09.247301 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.247274    2577 scope.go:117] "RemoveContainer" containerID="01fe0c81523ce519594a8bcef580db96c492cf6dc2cfd0733eac520660efce85"
Sep 17 00:35:09.247719 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.247400    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"
Sep 17 00:35:09.258543 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.258516    2577 scope.go:117] "RemoveContainer" containerID="01fe0c81523ce519594a8bcef580db96c492cf6dc2cfd0733eac520660efce85"
Sep 17 00:35:09.258881 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:35:09.258863    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01fe0c81523ce519594a8bcef580db96c492cf6dc2cfd0733eac520660efce85\": container with ID starting with 01fe0c81523ce519594a8bcef580db96c492cf6dc2cfd0733eac520660efce85 not found: ID does not exist" containerID="01fe0c81523ce519594a8bcef580db96c492cf6dc2cfd0733eac520660efce85"
Sep 17 00:35:09.258951 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.258891    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01fe0c81523ce519594a8bcef580db96c492cf6dc2cfd0733eac520660efce85"} err="failed to get container status \"01fe0c81523ce519594a8bcef580db96c492cf6dc2cfd0733eac520660efce85\": rpc error: code = NotFound desc = could not find container \"01fe0c81523ce519594a8bcef580db96c492cf6dc2cfd0733eac520660efce85\": container with ID starting with 01fe0c81523ce519594a8bcef580db96c492cf6dc2cfd0733eac520660efce85 not found: ID does not exist"
Sep 17 00:35:09.274121 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.274083    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"]
Sep 17 00:35:09.277865 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.277838    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2faea3/e2e-conflict-b-fe1bd81e-kserve-fc49f9d79-ph7bd"]
Sep 17 00:35:09.306564 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.306527    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-tls-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:35:09.306564 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.306557    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk9v8\" (UniqueName: \"kubernetes.io/projected/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-kube-api-access-kk9v8\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:35:09.306564 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.306568    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-dshm\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:35:09.306794 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.306580    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7-model-cache\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:35:09.608699 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:09.608614    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7" path="/var/lib/kubelet/pods/f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7/volumes"
Sep 17 00:35:14.320208 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:14.320160    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:35:14.320760 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:14.320317    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:35:14.332937 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:14.332900    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:35:15.294972 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:15.294937    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:35:26.807691 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:26.807655    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:35:30.023974 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:30.023939    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk"]
Sep 17 00:35:30.024406 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:30.024169    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk" podUID="b1a0712f-1bc8-4147-a844-c6aa86ef9fcf" containerName="maas-api" containerID="cri-o://16163bc4646a787f0d6ca5df87fc357a6e28cdeb737d765746b29e45726ec7cf" gracePeriod=30
Sep 17 00:35:30.278742 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:30.278672    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk"
Sep 17 00:35:30.368286 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:30.368255    2577 generic.go:356] "Generic (PLEG): container finished" podID="b1a0712f-1bc8-4147-a844-c6aa86ef9fcf" containerID="16163bc4646a787f0d6ca5df87fc357a6e28cdeb737d765746b29e45726ec7cf" exitCode=0
Sep 17 00:35:30.368506 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:30.368305    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk"
Sep 17 00:35:30.368506 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:30.368341    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk" event={"ID":"b1a0712f-1bc8-4147-a844-c6aa86ef9fcf","Type":"ContainerDied","Data":"16163bc4646a787f0d6ca5df87fc357a6e28cdeb737d765746b29e45726ec7cf"}
Sep 17 00:35:30.368506 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:30.368408    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk" event={"ID":"b1a0712f-1bc8-4147-a844-c6aa86ef9fcf","Type":"ContainerDied","Data":"94c1962e2684931a35a18d147623800f45dc154a5b8dc99307bdd243f7daea8f"}
Sep 17 00:35:30.368506 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:30.368484    2577 scope.go:117] "RemoveContainer" containerID="16163bc4646a787f0d6ca5df87fc357a6e28cdeb737d765746b29e45726ec7cf"
Sep 17 00:35:30.377819 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:30.377801    2577 scope.go:117] "RemoveContainer" containerID="16163bc4646a787f0d6ca5df87fc357a6e28cdeb737d765746b29e45726ec7cf"
Sep 17 00:35:30.378117 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:35:30.378098    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16163bc4646a787f0d6ca5df87fc357a6e28cdeb737d765746b29e45726ec7cf\": container with ID starting with 16163bc4646a787f0d6ca5df87fc357a6e28cdeb737d765746b29e45726ec7cf not found: ID does not exist" containerID="16163bc4646a787f0d6ca5df87fc357a6e28cdeb737d765746b29e45726ec7cf"
Sep 17 00:35:30.378187 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:30.378124    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16163bc4646a787f0d6ca5df87fc357a6e28cdeb737d765746b29e45726ec7cf"} err="failed to get container status \"16163bc4646a787f0d6ca5df87fc357a6e28cdeb737d765746b29e45726ec7cf\": rpc error: code = NotFound desc = could not find container \"16163bc4646a787f0d6ca5df87fc357a6e28cdeb737d765746b29e45726ec7cf\": container with ID starting with 16163bc4646a787f0d6ca5df87fc357a6e28cdeb737d765746b29e45726ec7cf not found: ID does not exist"
Sep 17 00:35:30.426688 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:30.426654    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf-metrics-certs\") pod \"b1a0712f-1bc8-4147-a844-c6aa86ef9fcf\" (UID: \"b1a0712f-1bc8-4147-a844-c6aa86ef9fcf\") "
Sep 17 00:35:30.426854 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:30.426749    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf-openshift-service-ca\") pod \"b1a0712f-1bc8-4147-a844-c6aa86ef9fcf\" (UID: \"b1a0712f-1bc8-4147-a844-c6aa86ef9fcf\") "
Sep 17 00:35:30.426854 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:30.426792    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz5xk\" (UniqueName: \"kubernetes.io/projected/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf-kube-api-access-fz5xk\") pod \"b1a0712f-1bc8-4147-a844-c6aa86ef9fcf\" (UID: \"b1a0712f-1bc8-4147-a844-c6aa86ef9fcf\") "
Sep 17 00:35:30.426854 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:30.426826    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf-maas-api-tls\") pod \"b1a0712f-1bc8-4147-a844-c6aa86ef9fcf\" (UID: \"b1a0712f-1bc8-4147-a844-c6aa86ef9fcf\") "
Sep 17 00:35:30.427144 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:30.427118    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b1a0712f-1bc8-4147-a844-c6aa86ef9fcf" (UID: "b1a0712f-1bc8-4147-a844-c6aa86ef9fcf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:35:30.428874 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:30.428838    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b1a0712f-1bc8-4147-a844-c6aa86ef9fcf" (UID: "b1a0712f-1bc8-4147-a844-c6aa86ef9fcf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:35:30.428978 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:30.428896    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b1a0712f-1bc8-4147-a844-c6aa86ef9fcf" (UID: "b1a0712f-1bc8-4147-a844-c6aa86ef9fcf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:35:30.428978 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:30.428911    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf-kube-api-access-fz5xk" (OuterVolumeSpecName: "kube-api-access-fz5xk") pod "b1a0712f-1bc8-4147-a844-c6aa86ef9fcf" (UID: "b1a0712f-1bc8-4147-a844-c6aa86ef9fcf"). InnerVolumeSpecName "kube-api-access-fz5xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:35:30.527793 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:30.527755    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf-openshift-service-ca\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:35:30.527793 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:30.527787    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz5xk\" (UniqueName: \"kubernetes.io/projected/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf-kube-api-access-fz5xk\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:35:30.527793 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:30.527797    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf-maas-api-tls\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:35:30.528029 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:30.527806    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf-metrics-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:35:30.690603 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:30.690573    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk"]
Sep 17 00:35:30.694270 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:30.694245    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c5ae0fdd-756fcfcdd-rsqsk"]
Sep 17 00:35:31.609459 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:31.609409    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1a0712f-1bc8-4147-a844-c6aa86ef9fcf" path="/var/lib/kubelet/pods/b1a0712f-1bc8-4147-a844-c6aa86ef9fcf/volumes"
Sep 17 00:35:38.540787 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:38.540747    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"]
Sep 17 00:35:52.407837 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:52.407799    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:35:53.560468 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:53.560387    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49" podUID="29ea83c6-01e8-4478-8ce0-ec50b6749561" containerName="main" containerID="cri-o://e4c0291dee39d17f4752455adb33c1f4cc20090a2284d6b57cc7575b8b1d88ee" gracePeriod=45
Sep 17 00:35:53.827017 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:53.826989    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:35:53.942657 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:53.942610    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29ea83c6-01e8-4478-8ce0-ec50b6749561-tmp-dir\") pod \"29ea83c6-01e8-4478-8ce0-ec50b6749561\" (UID: \"29ea83c6-01e8-4478-8ce0-ec50b6749561\") "
Sep 17 00:35:53.942657 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:53.942657    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29ea83c6-01e8-4478-8ce0-ec50b6749561-tls-certs\") pod \"29ea83c6-01e8-4478-8ce0-ec50b6749561\" (UID: \"29ea83c6-01e8-4478-8ce0-ec50b6749561\") "
Sep 17 00:35:53.942951 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:53.942699    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29ea83c6-01e8-4478-8ce0-ec50b6749561-home\") pod \"29ea83c6-01e8-4478-8ce0-ec50b6749561\" (UID: \"29ea83c6-01e8-4478-8ce0-ec50b6749561\") "
Sep 17 00:35:53.942951 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:53.942716    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzrkd\" (UniqueName: \"kubernetes.io/projected/29ea83c6-01e8-4478-8ce0-ec50b6749561-kube-api-access-bzrkd\") pod \"29ea83c6-01e8-4478-8ce0-ec50b6749561\" (UID: \"29ea83c6-01e8-4478-8ce0-ec50b6749561\") "
Sep 17 00:35:53.942951 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:53.942741    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29ea83c6-01e8-4478-8ce0-ec50b6749561-model-cache\") pod \"29ea83c6-01e8-4478-8ce0-ec50b6749561\" (UID: \"29ea83c6-01e8-4478-8ce0-ec50b6749561\") "
Sep 17 00:35:53.942951 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:53.942829    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29ea83c6-01e8-4478-8ce0-ec50b6749561-dshm\") pod \"29ea83c6-01e8-4478-8ce0-ec50b6749561\" (UID: \"29ea83c6-01e8-4478-8ce0-ec50b6749561\") "
Sep 17 00:35:53.943161 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:53.942922    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ea83c6-01e8-4478-8ce0-ec50b6749561-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "29ea83c6-01e8-4478-8ce0-ec50b6749561" (UID: "29ea83c6-01e8-4478-8ce0-ec50b6749561"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:35:53.943161 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:53.942970    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ea83c6-01e8-4478-8ce0-ec50b6749561-home" (OuterVolumeSpecName: "home") pod "29ea83c6-01e8-4478-8ce0-ec50b6749561" (UID: "29ea83c6-01e8-4478-8ce0-ec50b6749561"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:35:53.943161 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:53.943064    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ea83c6-01e8-4478-8ce0-ec50b6749561-model-cache" (OuterVolumeSpecName: "model-cache") pod "29ea83c6-01e8-4478-8ce0-ec50b6749561" (UID: "29ea83c6-01e8-4478-8ce0-ec50b6749561"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:35:53.943309 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:53.943172    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29ea83c6-01e8-4478-8ce0-ec50b6749561-tmp-dir\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:35:53.943309 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:53.943186    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29ea83c6-01e8-4478-8ce0-ec50b6749561-home\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:35:53.943309 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:53.943195    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29ea83c6-01e8-4478-8ce0-ec50b6749561-model-cache\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:35:53.945164 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:53.945134    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ea83c6-01e8-4478-8ce0-ec50b6749561-dshm" (OuterVolumeSpecName: "dshm") pod "29ea83c6-01e8-4478-8ce0-ec50b6749561" (UID: "29ea83c6-01e8-4478-8ce0-ec50b6749561"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:35:53.945271 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:53.945138    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29ea83c6-01e8-4478-8ce0-ec50b6749561-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29ea83c6-01e8-4478-8ce0-ec50b6749561" (UID: "29ea83c6-01e8-4478-8ce0-ec50b6749561"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:35:53.945316 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:53.945261    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ea83c6-01e8-4478-8ce0-ec50b6749561-kube-api-access-bzrkd" (OuterVolumeSpecName: "kube-api-access-bzrkd") pod "29ea83c6-01e8-4478-8ce0-ec50b6749561" (UID: "29ea83c6-01e8-4478-8ce0-ec50b6749561"). InnerVolumeSpecName "kube-api-access-bzrkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:35:54.044215 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:54.044159    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29ea83c6-01e8-4478-8ce0-ec50b6749561-tls-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:35:54.044215 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:54.044209    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzrkd\" (UniqueName: \"kubernetes.io/projected/29ea83c6-01e8-4478-8ce0-ec50b6749561-kube-api-access-bzrkd\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:35:54.044215 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:54.044222    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29ea83c6-01e8-4478-8ce0-ec50b6749561-dshm\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:35:54.514944 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:54.514909    2577 generic.go:356] "Generic (PLEG): container finished" podID="29ea83c6-01e8-4478-8ce0-ec50b6749561" containerID="e4c0291dee39d17f4752455adb33c1f4cc20090a2284d6b57cc7575b8b1d88ee" exitCode=0
Sep 17 00:35:54.515155 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:54.514956    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"
Sep 17 00:35:54.515155 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:54.515002    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49" event={"ID":"29ea83c6-01e8-4478-8ce0-ec50b6749561","Type":"ContainerDied","Data":"e4c0291dee39d17f4752455adb33c1f4cc20090a2284d6b57cc7575b8b1d88ee"}
Sep 17 00:35:54.515155 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:54.515052    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49" event={"ID":"29ea83c6-01e8-4478-8ce0-ec50b6749561","Type":"ContainerDied","Data":"8f04c546e9a0aca2b6ece385829f2bf824f0800567989ae504485a2b5fab14d8"}
Sep 17 00:35:54.515155 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:54.515076    2577 scope.go:117] "RemoveContainer" containerID="e4c0291dee39d17f4752455adb33c1f4cc20090a2284d6b57cc7575b8b1d88ee"
Sep 17 00:35:54.525583 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:54.525564    2577 scope.go:117] "RemoveContainer" containerID="e4c0291dee39d17f4752455adb33c1f4cc20090a2284d6b57cc7575b8b1d88ee"
Sep 17 00:35:54.525937 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:35:54.525916    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4c0291dee39d17f4752455adb33c1f4cc20090a2284d6b57cc7575b8b1d88ee\": container with ID starting with e4c0291dee39d17f4752455adb33c1f4cc20090a2284d6b57cc7575b8b1d88ee not found: ID does not exist" containerID="e4c0291dee39d17f4752455adb33c1f4cc20090a2284d6b57cc7575b8b1d88ee"
Sep 17 00:35:54.526078 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:54.525942    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4c0291dee39d17f4752455adb33c1f4cc20090a2284d6b57cc7575b8b1d88ee"} err="failed to get container status \"e4c0291dee39d17f4752455adb33c1f4cc20090a2284d6b57cc7575b8b1d88ee\": rpc error: code = NotFound desc = could not find container \"e4c0291dee39d17f4752455adb33c1f4cc20090a2284d6b57cc7575b8b1d88ee\": container with ID starting with e4c0291dee39d17f4752455adb33c1f4cc20090a2284d6b57cc7575b8b1d88ee not found: ID does not exist"
Sep 17 00:35:54.539439 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:54.539349    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"]
Sep 17 00:35:54.542815 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:54.542789    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-c5ae0fdd/e2e-lifecycle-model-c5ae0fdd-kserve-74f54b5bf9-vwx49"]
Sep 17 00:35:55.607823 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:35:55.607793    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29ea83c6-01e8-4478-8ce0-ec50b6749561" path="/var/lib/kubelet/pods/29ea83c6-01e8-4478-8ce0-ec50b6749561/volumes"
Sep 17 00:36:22.899870 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:22.899834    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:36:26.295091 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.295049    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"]
Sep 17 00:36:26.295687 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.295663    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29ea83c6-01e8-4478-8ce0-ec50b6749561" containerName="main"
Sep 17 00:36:26.295768 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.295688    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="29ea83c6-01e8-4478-8ce0-ec50b6749561" containerName="main"
Sep 17 00:36:26.295768 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.295727    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7" containerName="main"
Sep 17 00:36:26.295768 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.295735    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7" containerName="main"
Sep 17 00:36:26.295768 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.295752    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1a0712f-1bc8-4147-a844-c6aa86ef9fcf" containerName="maas-api"
Sep 17 00:36:26.295768 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.295762    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1a0712f-1bc8-4147-a844-c6aa86ef9fcf" containerName="maas-api"
Sep 17 00:36:26.296037 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.295861    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1a0712f-1bc8-4147-a844-c6aa86ef9fcf" containerName="maas-api"
Sep 17 00:36:26.296037 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.295877    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6a2a3a1-31b6-4a4b-a2ad-143701b41dc7" containerName="main"
Sep 17 00:36:26.296037 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.295886    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="29ea83c6-01e8-4478-8ce0-ec50b6749561" containerName="main"
Sep 17 00:36:26.299467 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.299444    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:36:26.302225 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.302197    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2faea3\"/\"distinct-2-w4-2faea3-kserve-self-signed-certs\""
Sep 17 00:36:26.308688 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.308661    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"]
Sep 17 00:36:26.358653 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.358612    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btbjg\" (UniqueName: \"kubernetes.io/projected/1886bc33-36ad-492e-a0c5-a2a317988c34-kube-api-access-btbjg\") pod \"distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6\" (UID: \"1886bc33-36ad-492e-a0c5-a2a317988c34\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:36:26.358851 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.358681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1886bc33-36ad-492e-a0c5-a2a317988c34-tmp-dir\") pod \"distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6\" (UID: \"1886bc33-36ad-492e-a0c5-a2a317988c34\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:36:26.358851 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.358753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1886bc33-36ad-492e-a0c5-a2a317988c34-home\") pod \"distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6\" (UID: \"1886bc33-36ad-492e-a0c5-a2a317988c34\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:36:26.358851 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.358796    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1886bc33-36ad-492e-a0c5-a2a317988c34-dshm\") pod \"distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6\" (UID: \"1886bc33-36ad-492e-a0c5-a2a317988c34\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:36:26.358851 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.358830    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1886bc33-36ad-492e-a0c5-a2a317988c34-tls-certs\") pod \"distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6\" (UID: \"1886bc33-36ad-492e-a0c5-a2a317988c34\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:36:26.359064 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.358941    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1886bc33-36ad-492e-a0c5-a2a317988c34-model-cache\") pod \"distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6\" (UID: \"1886bc33-36ad-492e-a0c5-a2a317988c34\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:36:26.460379 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.460331    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1886bc33-36ad-492e-a0c5-a2a317988c34-home\") pod \"distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6\" (UID: \"1886bc33-36ad-492e-a0c5-a2a317988c34\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:36:26.460379 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.460382    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1886bc33-36ad-492e-a0c5-a2a317988c34-dshm\") pod \"distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6\" (UID: \"1886bc33-36ad-492e-a0c5-a2a317988c34\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:36:26.460671 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.460529    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1886bc33-36ad-492e-a0c5-a2a317988c34-tls-certs\") pod \"distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6\" (UID: \"1886bc33-36ad-492e-a0c5-a2a317988c34\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:36:26.460671 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.460616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1886bc33-36ad-492e-a0c5-a2a317988c34-model-cache\") pod \"distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6\" (UID: \"1886bc33-36ad-492e-a0c5-a2a317988c34\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:36:26.460788 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.460679    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btbjg\" (UniqueName: \"kubernetes.io/projected/1886bc33-36ad-492e-a0c5-a2a317988c34-kube-api-access-btbjg\") pod \"distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6\" (UID: \"1886bc33-36ad-492e-a0c5-a2a317988c34\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:36:26.460788 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.460731    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1886bc33-36ad-492e-a0c5-a2a317988c34-tmp-dir\") pod \"distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6\" (UID: \"1886bc33-36ad-492e-a0c5-a2a317988c34\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:36:26.460927 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.460900    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1886bc33-36ad-492e-a0c5-a2a317988c34-home\") pod \"distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6\" (UID: \"1886bc33-36ad-492e-a0c5-a2a317988c34\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:36:26.460980 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.460943    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1886bc33-36ad-492e-a0c5-a2a317988c34-model-cache\") pod \"distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6\" (UID: \"1886bc33-36ad-492e-a0c5-a2a317988c34\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:36:26.461070 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.461050    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1886bc33-36ad-492e-a0c5-a2a317988c34-tmp-dir\") pod \"distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6\" (UID: \"1886bc33-36ad-492e-a0c5-a2a317988c34\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:36:26.462817 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.462793    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1886bc33-36ad-492e-a0c5-a2a317988c34-dshm\") pod \"distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6\" (UID: \"1886bc33-36ad-492e-a0c5-a2a317988c34\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:36:26.463027 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.463009    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1886bc33-36ad-492e-a0c5-a2a317988c34-tls-certs\") pod \"distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6\" (UID: \"1886bc33-36ad-492e-a0c5-a2a317988c34\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:36:26.468738 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.468711    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btbjg\" (UniqueName: \"kubernetes.io/projected/1886bc33-36ad-492e-a0c5-a2a317988c34-kube-api-access-btbjg\") pod \"distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6\" (UID: \"1886bc33-36ad-492e-a0c5-a2a317988c34\") " pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:36:26.609649 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.609553    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:36:26.883921 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:26.883893    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"]
Sep 17 00:36:26.885732 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:36:26.885700    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1886bc33_36ad_492e_a0c5_a2a317988c34.slice/crio-cf9105b273ed5fb16788608a94542a76dd2823d6a8833af90e0f7fa93db9c118 WatchSource:0}: Error finding container cf9105b273ed5fb16788608a94542a76dd2823d6a8833af90e0f7fa93db9c118: Status 404 returned error can't find the container with id cf9105b273ed5fb16788608a94542a76dd2823d6a8833af90e0f7fa93db9c118
Sep 17 00:36:27.708918 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:27.708878    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6" event={"ID":"1886bc33-36ad-492e-a0c5-a2a317988c34","Type":"ContainerStarted","Data":"aad81999c131fffef22c07c020895981989aa4c6fce7e8d5507eceef729a33ce"}
Sep 17 00:36:27.709303 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:27.708928    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6" event={"ID":"1886bc33-36ad-492e-a0c5-a2a317988c34","Type":"ContainerStarted","Data":"cf9105b273ed5fb16788608a94542a76dd2823d6a8833af90e0f7fa93db9c118"}
Sep 17 00:36:27.725900 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:27.725840    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6" podStartSLOduration=1.7258223830000001 podStartE2EDuration="1.725822383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:36:27.724035842 +0000 UTC m=+1600.582528980" watchObservedRunningTime="2026-09-17 00:36:27.725822383 +0000 UTC m=+1600.584315518"
Sep 17 00:36:35.502996 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.502960    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"]
Sep 17 00:36:35.506531 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.506508    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:36:35.509210 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.509183    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-07f9f2\"/\"openshift-service-ca.crt\""
Sep 17 00:36:35.511150 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.510884    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-07f9f2\"/\"auth-test-model-07f9f2-kserve-self-signed-certs\""
Sep 17 00:36:35.511150 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.510954    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-07f9f2\"/\"default-dockercfg-tp66z\""
Sep 17 00:36:35.512054 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.512031    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-07f9f2\"/\"kube-root-ca.crt\""
Sep 17 00:36:35.520229 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.520197    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"]
Sep 17 00:36:35.545863 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.545826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/602e7a60-9b84-4f02-8595-db8827604955-tls-certs\") pod \"auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp\" (UID: \"602e7a60-9b84-4f02-8595-db8827604955\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:36:35.546047 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.545887    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/602e7a60-9b84-4f02-8595-db8827604955-model-cache\") pod \"auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp\" (UID: \"602e7a60-9b84-4f02-8595-db8827604955\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:36:35.546047 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.545905    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/602e7a60-9b84-4f02-8595-db8827604955-dshm\") pod \"auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp\" (UID: \"602e7a60-9b84-4f02-8595-db8827604955\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:36:35.546047 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.545993    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szpjf\" (UniqueName: \"kubernetes.io/projected/602e7a60-9b84-4f02-8595-db8827604955-kube-api-access-szpjf\") pod \"auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp\" (UID: \"602e7a60-9b84-4f02-8595-db8827604955\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:36:35.546209 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.546068    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/602e7a60-9b84-4f02-8595-db8827604955-tmp-dir\") pod \"auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp\" (UID: \"602e7a60-9b84-4f02-8595-db8827604955\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:36:35.546209 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.546101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/602e7a60-9b84-4f02-8595-db8827604955-home\") pod \"auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp\" (UID: \"602e7a60-9b84-4f02-8595-db8827604955\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:36:35.647398 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.647360    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/602e7a60-9b84-4f02-8595-db8827604955-model-cache\") pod \"auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp\" (UID: \"602e7a60-9b84-4f02-8595-db8827604955\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:36:35.647615 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.647407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/602e7a60-9b84-4f02-8595-db8827604955-dshm\") pod \"auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp\" (UID: \"602e7a60-9b84-4f02-8595-db8827604955\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:36:35.647615 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.647463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szpjf\" (UniqueName: \"kubernetes.io/projected/602e7a60-9b84-4f02-8595-db8827604955-kube-api-access-szpjf\") pod \"auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp\" (UID: \"602e7a60-9b84-4f02-8595-db8827604955\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:36:35.647615 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.647518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/602e7a60-9b84-4f02-8595-db8827604955-tmp-dir\") pod \"auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp\" (UID: \"602e7a60-9b84-4f02-8595-db8827604955\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:36:35.647615 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.647547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/602e7a60-9b84-4f02-8595-db8827604955-home\") pod \"auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp\" (UID: \"602e7a60-9b84-4f02-8595-db8827604955\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:36:35.647845 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.647665    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/602e7a60-9b84-4f02-8595-db8827604955-tls-certs\") pod \"auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp\" (UID: \"602e7a60-9b84-4f02-8595-db8827604955\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:36:35.647900 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.647871    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/602e7a60-9b84-4f02-8595-db8827604955-model-cache\") pod \"auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp\" (UID: \"602e7a60-9b84-4f02-8595-db8827604955\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:36:35.648267 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.647971    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/602e7a60-9b84-4f02-8595-db8827604955-home\") pod \"auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp\" (UID: \"602e7a60-9b84-4f02-8595-db8827604955\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:36:35.648267 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.648225    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/602e7a60-9b84-4f02-8595-db8827604955-tmp-dir\") pod \"auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp\" (UID: \"602e7a60-9b84-4f02-8595-db8827604955\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:36:35.649930 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.649907    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/602e7a60-9b84-4f02-8595-db8827604955-dshm\") pod \"auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp\" (UID: \"602e7a60-9b84-4f02-8595-db8827604955\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:36:35.650437 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.650401    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/602e7a60-9b84-4f02-8595-db8827604955-tls-certs\") pod \"auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp\" (UID: \"602e7a60-9b84-4f02-8595-db8827604955\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:36:35.655181 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.655155    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szpjf\" (UniqueName: \"kubernetes.io/projected/602e7a60-9b84-4f02-8595-db8827604955-kube-api-access-szpjf\") pod \"auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp\" (UID: \"602e7a60-9b84-4f02-8595-db8827604955\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:36:35.820309 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:35.820203    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:36:36.097763 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:36.097729    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"]
Sep 17 00:36:36.099339 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:36:36.099312    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod602e7a60_9b84_4f02_8595_db8827604955.slice/crio-744ea17c1aea68b790db6ba07812e805fe837fe2d4662710c84b64d1b0013fd8 WatchSource:0}: Error finding container 744ea17c1aea68b790db6ba07812e805fe837fe2d4662710c84b64d1b0013fd8: Status 404 returned error can't find the container with id 744ea17c1aea68b790db6ba07812e805fe837fe2d4662710c84b64d1b0013fd8
Sep 17 00:36:36.610161 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:36.610126    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:36:36.610687 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:36.610179    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:36:36.623585 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:36.623553    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:36:36.763195 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:36.763143    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp" event={"ID":"602e7a60-9b84-4f02-8595-db8827604955","Type":"ContainerStarted","Data":"202f74b7d899d249e94b455c675d88caa4495d8dcb9942235380769ce18e8d42"}
Sep 17 00:36:36.763195 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:36.763199    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp" event={"ID":"602e7a60-9b84-4f02-8595-db8827604955","Type":"ContainerStarted","Data":"744ea17c1aea68b790db6ba07812e805fe837fe2d4662710c84b64d1b0013fd8"}
Sep 17 00:36:36.775041 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:36.775007    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:36:36.780980 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:36.780932    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp" podStartSLOduration=1.780917031 podStartE2EDuration="1.780917031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:36:36.778370258 +0000 UTC m=+1609.636863394" watchObservedRunningTime="2026-09-17 00:36:36.780917031 +0000 UTC m=+1609.639410166"
Sep 17 00:36:45.820476 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:45.820374    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:36:45.820476 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:45.820456    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:36:45.841127 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:45.841088    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:36:46.831530 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:46.831499    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:36:47.799297 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:36:47.799265    2577 scope.go:117] "RemoveContainer" containerID="627b38c35bbabe11289fd8b49da8ef0c8b17d484b78be63d6558d60769ecf0bf"
Sep 17 00:37:07.804732 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:37:07.804693    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:37:54.807870 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:37:54.807832    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:38:10.301984 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:10.301946    2577 generic.go:356] "Generic (PLEG): container finished" podID="b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969" containerID="2f0c6449ce251be029d06288ee0c364626ea75e57c1be1a3339920f930194397" exitCode=137
Sep 17 00:38:10.302465 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:10.302018    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg" event={"ID":"b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969","Type":"ContainerDied","Data":"2f0c6449ce251be029d06288ee0c364626ea75e57c1be1a3339920f930194397"}
Sep 17 00:38:10.302465 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:10.302379    2577 scope.go:117] "RemoveContainer" containerID="2f0c6449ce251be029d06288ee0c364626ea75e57c1be1a3339920f930194397"
Sep 17 00:38:11.310285 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:11.310241    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg" event={"ID":"b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969","Type":"ContainerStarted","Data":"ee7032d0f71608edb5109a82bab6c54e8e3fe94be0188600bfef1b87c8a63b14"}
Sep 17 00:38:11.310814 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:11.310614    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg"
Sep 17 00:38:14.301038 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:14.301004    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"]
Sep 17 00:38:14.304563 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:14.304540    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"
Sep 17 00:38:14.307102 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:14.307075    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1b3aae-serving-cert\""
Sep 17 00:38:14.315750 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:14.315726    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"]
Sep 17 00:38:14.407190 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:14.407152    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-openshift-service-ca\") pod \"maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb\" (UID: \"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"
Sep 17 00:38:14.407360 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:14.407223    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-maas-api-tls\") pod \"maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb\" (UID: \"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"
Sep 17 00:38:14.407360 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:14.407257    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lm7l\" (UniqueName: \"kubernetes.io/projected/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-kube-api-access-8lm7l\") pod \"maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb\" (UID: \"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"
Sep 17 00:38:14.407360 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:14.407280    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-metrics-certs\") pod \"maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb\" (UID: \"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"
Sep 17 00:38:14.508666 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:14.508623    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-openshift-service-ca\") pod \"maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb\" (UID: \"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"
Sep 17 00:38:14.508817 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:14.508702    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-maas-api-tls\") pod \"maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb\" (UID: \"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"
Sep 17 00:38:14.508817 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:14.508735    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lm7l\" (UniqueName: \"kubernetes.io/projected/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-kube-api-access-8lm7l\") pod \"maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb\" (UID: \"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"
Sep 17 00:38:14.508817 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:14.508770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-metrics-certs\") pod \"maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb\" (UID: \"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"
Sep 17 00:38:14.508936 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:38:14.508858    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-serving-cert: secret "maas-api-e2e-api-b-1b3aae-serving-cert" not found
Sep 17 00:38:14.508976 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:38:14.508942    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-maas-api-tls podName:f6ac5c53-01a8-431c-bd96-e8e6ebe5490b nodeName:}" failed. No retries permitted until 2026-09-17 00:38:15.008923468 +0000 UTC m=+1707.867416584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-maas-api-tls") pod "maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb" (UID: "f6ac5c53-01a8-431c-bd96-e8e6ebe5490b") : secret "maas-api-e2e-api-b-1b3aae-serving-cert" not found
Sep 17 00:38:14.509462 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:14.509407    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-openshift-service-ca\") pod \"maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb\" (UID: \"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"
Sep 17 00:38:14.511399 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:14.511378    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-metrics-certs\") pod \"maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb\" (UID: \"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"
Sep 17 00:38:14.516864 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:14.516846    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lm7l\" (UniqueName: \"kubernetes.io/projected/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-kube-api-access-8lm7l\") pod \"maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb\" (UID: \"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"
Sep 17 00:38:15.015284 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:15.015246    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-maas-api-tls\") pod \"maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb\" (UID: \"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"
Sep 17 00:38:15.017891 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:15.017857    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-maas-api-tls\") pod \"maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb\" (UID: \"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"
Sep 17 00:38:15.217017 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:15.216973    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"
Sep 17 00:38:15.497085 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:15.497046    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"]
Sep 17 00:38:15.498377 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:38:15.498351    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ac5c53_01a8_431c_bd96_e8e6ebe5490b.slice/crio-a9ac99f1e5f765d175368b95f95f6d77e25d41d5bd6098783415908c8990545d WatchSource:0}: Error finding container a9ac99f1e5f765d175368b95f95f6d77e25d41d5bd6098783415908c8990545d: Status 404 returned error can't find the container with id a9ac99f1e5f765d175368b95f95f6d77e25d41d5bd6098783415908c8990545d
Sep 17 00:38:15.499665 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:15.499647    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 00:38:16.343747 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:16.343709    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb" event={"ID":"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b","Type":"ContainerStarted","Data":"e333f96574691b3582d387c662325cbded2a4df9246f1981d4ac46bf30572d9b"}
Sep 17 00:38:16.343936 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:16.343764    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb" event={"ID":"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b","Type":"ContainerStarted","Data":"a9ac99f1e5f765d175368b95f95f6d77e25d41d5bd6098783415908c8990545d"}
Sep 17 00:38:16.343936 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:16.343811    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"
Sep 17 00:38:16.362530 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:16.362474    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb" podStartSLOduration=2.020195257 podStartE2EDuration="2.362452501s" totalImagesPullingTime="342.257244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:38:16.359929791 +0000 UTC m=+1709.218422927" watchObservedRunningTime="2026-09-17 00:38:16.362452501 +0000 UTC m=+1709.220945614"
Sep 17 00:38:22.319684 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:22.319648    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg"
Sep 17 00:38:22.356812 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:22.356782    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"
Sep 17 00:38:39.403965 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:38:39.403921    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:39:06.864459 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:06.864409    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"]
Sep 17 00:39:06.880933 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:06.880899    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"]
Sep 17 00:39:06.895730 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:06.895686    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"]
Sep 17 00:39:08.810480 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:08.810406    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:39:17.793315 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:17.793280    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cfe049-29826750-ltqkj"]
Sep 17 00:39:17.796834 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:17.796807    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-cfe049-29826750-ltqkj"]
Sep 17 00:39:18.111859 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:18.111775    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"]
Sep 17 00:39:18.112110 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:18.112069    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb" podUID="f6ac5c53-01a8-431c-bd96-e8e6ebe5490b" containerName="maas-api" containerID="cri-o://e333f96574691b3582d387c662325cbded2a4df9246f1981d4ac46bf30572d9b" gracePeriod=30
Sep 17 00:39:18.364793 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:18.364731    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"
Sep 17 00:39:18.427252 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:18.427223    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-maas-api-tls\") pod \"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b\" (UID: \"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b\") "
Sep 17 00:39:18.427444 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:18.427263    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-metrics-certs\") pod \"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b\" (UID: \"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b\") "
Sep 17 00:39:18.427444 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:18.427335    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lm7l\" (UniqueName: \"kubernetes.io/projected/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-kube-api-access-8lm7l\") pod \"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b\" (UID: \"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b\") "
Sep 17 00:39:18.427444 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:18.427361    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-openshift-service-ca\") pod \"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b\" (UID: \"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b\") "
Sep 17 00:39:18.427791 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:18.427767    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f6ac5c53-01a8-431c-bd96-e8e6ebe5490b" (UID: "f6ac5c53-01a8-431c-bd96-e8e6ebe5490b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:39:18.429325 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:18.429299    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f6ac5c53-01a8-431c-bd96-e8e6ebe5490b" (UID: "f6ac5c53-01a8-431c-bd96-e8e6ebe5490b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:39:18.429465 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:18.429386    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-kube-api-access-8lm7l" (OuterVolumeSpecName: "kube-api-access-8lm7l") pod "f6ac5c53-01a8-431c-bd96-e8e6ebe5490b" (UID: "f6ac5c53-01a8-431c-bd96-e8e6ebe5490b"). InnerVolumeSpecName "kube-api-access-8lm7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:39:18.429465 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:18.429440    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f6ac5c53-01a8-431c-bd96-e8e6ebe5490b" (UID: "f6ac5c53-01a8-431c-bd96-e8e6ebe5490b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:39:18.529243 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:18.529208    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-maas-api-tls\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:18.529243 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:18.529236    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-metrics-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:18.529243 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:18.529246    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lm7l\" (UniqueName: \"kubernetes.io/projected/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-kube-api-access-8lm7l\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:18.529508 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:18.529255    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b-openshift-service-ca\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:18.705156 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:18.705121    2577 generic.go:356] "Generic (PLEG): container finished" podID="f6ac5c53-01a8-431c-bd96-e8e6ebe5490b" containerID="e333f96574691b3582d387c662325cbded2a4df9246f1981d4ac46bf30572d9b" exitCode=0
Sep 17 00:39:18.705320 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:18.705168    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"
Sep 17 00:39:18.705320 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:18.705195    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb" event={"ID":"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b","Type":"ContainerDied","Data":"e333f96574691b3582d387c662325cbded2a4df9246f1981d4ac46bf30572d9b"}
Sep 17 00:39:18.705320 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:18.705236    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb" event={"ID":"f6ac5c53-01a8-431c-bd96-e8e6ebe5490b","Type":"ContainerDied","Data":"a9ac99f1e5f765d175368b95f95f6d77e25d41d5bd6098783415908c8990545d"}
Sep 17 00:39:18.705320 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:18.705257    2577 scope.go:117] "RemoveContainer" containerID="e333f96574691b3582d387c662325cbded2a4df9246f1981d4ac46bf30572d9b"
Sep 17 00:39:18.715331 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:18.715313    2577 scope.go:117] "RemoveContainer" containerID="e333f96574691b3582d387c662325cbded2a4df9246f1981d4ac46bf30572d9b"
Sep 17 00:39:18.715609 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:39:18.715587    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e333f96574691b3582d387c662325cbded2a4df9246f1981d4ac46bf30572d9b\": container with ID starting with e333f96574691b3582d387c662325cbded2a4df9246f1981d4ac46bf30572d9b not found: ID does not exist" containerID="e333f96574691b3582d387c662325cbded2a4df9246f1981d4ac46bf30572d9b"
Sep 17 00:39:18.715705 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:18.715615    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e333f96574691b3582d387c662325cbded2a4df9246f1981d4ac46bf30572d9b"} err="failed to get container status \"e333f96574691b3582d387c662325cbded2a4df9246f1981d4ac46bf30572d9b\": rpc error: code = NotFound desc = could not find container \"e333f96574691b3582d387c662325cbded2a4df9246f1981d4ac46bf30572d9b\": container with ID starting with e333f96574691b3582d387c662325cbded2a4df9246f1981d4ac46bf30572d9b not found: ID does not exist"
Sep 17 00:39:18.728786 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:18.728762    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"]
Sep 17 00:39:18.731022 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:18.731010    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1b3aae-57f949d5bf-pgvxb"]
Sep 17 00:39:19.607911 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:19.607879    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="622b3f7f-3b04-460d-898f-9a8935125a56" path="/var/lib/kubelet/pods/622b3f7f-3b04-460d-898f-9a8935125a56/volumes"
Sep 17 00:39:19.608284 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:19.608270    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6ac5c53-01a8-431c-bd96-e8e6ebe5490b" path="/var/lib/kubelet/pods/f6ac5c53-01a8-431c-bd96-e8e6ebe5490b/volumes"
Sep 17 00:39:21.885584 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:21.885542    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf" podUID="a2455bef-2bec-4e2e-9aed-f3734c47b8c9" containerName="main" containerID="cri-o://d0e41ead288ee83216691e320f8af73ac8812d00a6452faf59845111921d8595" gracePeriod=45
Sep 17 00:39:21.908350 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:21.908289    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r" podUID="0f93433d-4c9e-40e1-8108-410cee61d3eb" containerName="main" containerID="cri-o://9b7a108b97fe2af9a0871cb563e76c5ab1ffb25ec64a116b3330bd29d559e4a5" gracePeriod=45
Sep 17 00:39:21.915145 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:21.915091    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8" podUID="97a1c787-acbc-4d1c-9657-b79b9db6de85" containerName="main" containerID="cri-o://7baf95da4bc3e9f9df760a9b8d92891f81a2eb5aff8267189512ba7ac7720bf0" gracePeriod=45
Sep 17 00:39:22.165221 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.165193    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:39:22.218798 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.218771    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:39:22.222681 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.222653    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:39:22.263467 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.263432    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-tmp-dir\") pod \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\" (UID: \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\") "
Sep 17 00:39:22.263643 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.263475    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f93433d-4c9e-40e1-8108-410cee61d3eb-home\") pod \"0f93433d-4c9e-40e1-8108-410cee61d3eb\" (UID: \"0f93433d-4c9e-40e1-8108-410cee61d3eb\") "
Sep 17 00:39:22.263643 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.263498    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-tls-certs\") pod \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\" (UID: \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\") "
Sep 17 00:39:22.263643 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.263536    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqcxf\" (UniqueName: \"kubernetes.io/projected/97a1c787-acbc-4d1c-9657-b79b9db6de85-kube-api-access-nqcxf\") pod \"97a1c787-acbc-4d1c-9657-b79b9db6de85\" (UID: \"97a1c787-acbc-4d1c-9657-b79b9db6de85\") "
Sep 17 00:39:22.263643 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.263559    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a1c787-acbc-4d1c-9657-b79b9db6de85-tmp-dir\") pod \"97a1c787-acbc-4d1c-9657-b79b9db6de85\" (UID: \"97a1c787-acbc-4d1c-9657-b79b9db6de85\") "
Sep 17 00:39:22.263643 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.263587    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a1c787-acbc-4d1c-9657-b79b9db6de85-dshm\") pod \"97a1c787-acbc-4d1c-9657-b79b9db6de85\" (UID: \"97a1c787-acbc-4d1c-9657-b79b9db6de85\") "
Sep 17 00:39:22.263917 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.263655    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f93433d-4c9e-40e1-8108-410cee61d3eb-model-cache\") pod \"0f93433d-4c9e-40e1-8108-410cee61d3eb\" (UID: \"0f93433d-4c9e-40e1-8108-410cee61d3eb\") "
Sep 17 00:39:22.263917 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.263689    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-home\") pod \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\" (UID: \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\") "
Sep 17 00:39:22.263917 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.263722    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f93433d-4c9e-40e1-8108-410cee61d3eb-home" (OuterVolumeSpecName: "home") pod "0f93433d-4c9e-40e1-8108-410cee61d3eb" (UID: "0f93433d-4c9e-40e1-8108-410cee61d3eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:39:22.263917 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.263746    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a1c787-acbc-4d1c-9657-b79b9db6de85-tls-certs\") pod \"97a1c787-acbc-4d1c-9657-b79b9db6de85\" (UID: \"97a1c787-acbc-4d1c-9657-b79b9db6de85\") "
Sep 17 00:39:22.263917 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.263751    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2455bef-2bec-4e2e-9aed-f3734c47b8c9" (UID: "a2455bef-2bec-4e2e-9aed-f3734c47b8c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:39:22.263917 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.263785    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a1c787-acbc-4d1c-9657-b79b9db6de85-model-cache\") pod \"97a1c787-acbc-4d1c-9657-b79b9db6de85\" (UID: \"97a1c787-acbc-4d1c-9657-b79b9db6de85\") "
Sep 17 00:39:22.263917 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.263820    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f93433d-4c9e-40e1-8108-410cee61d3eb-dshm\") pod \"0f93433d-4c9e-40e1-8108-410cee61d3eb\" (UID: \"0f93433d-4c9e-40e1-8108-410cee61d3eb\") "
Sep 17 00:39:22.263917 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.263846    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f93433d-4c9e-40e1-8108-410cee61d3eb-tls-certs\") pod \"0f93433d-4c9e-40e1-8108-410cee61d3eb\" (UID: \"0f93433d-4c9e-40e1-8108-410cee61d3eb\") "
Sep 17 00:39:22.263917 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.263844    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a1c787-acbc-4d1c-9657-b79b9db6de85-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "97a1c787-acbc-4d1c-9657-b79b9db6de85" (UID: "97a1c787-acbc-4d1c-9657-b79b9db6de85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:39:22.263917 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.263870    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a1c787-acbc-4d1c-9657-b79b9db6de85-home\") pod \"97a1c787-acbc-4d1c-9657-b79b9db6de85\" (UID: \"97a1c787-acbc-4d1c-9657-b79b9db6de85\") "
Sep 17 00:39:22.263917 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.263902    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm2r6\" (UniqueName: \"kubernetes.io/projected/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-kube-api-access-zm2r6\") pod \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\" (UID: \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\") "
Sep 17 00:39:22.264480 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.263937    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tc7f\" (UniqueName: \"kubernetes.io/projected/0f93433d-4c9e-40e1-8108-410cee61d3eb-kube-api-access-9tc7f\") pod \"0f93433d-4c9e-40e1-8108-410cee61d3eb\" (UID: \"0f93433d-4c9e-40e1-8108-410cee61d3eb\") "
Sep 17 00:39:22.264480 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.263974    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-dshm\") pod \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\" (UID: \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\") "
Sep 17 00:39:22.264480 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.264022    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f93433d-4c9e-40e1-8108-410cee61d3eb-tmp-dir\") pod \"0f93433d-4c9e-40e1-8108-410cee61d3eb\" (UID: \"0f93433d-4c9e-40e1-8108-410cee61d3eb\") "
Sep 17 00:39:22.264480 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.264049    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-model-cache\") pod \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\" (UID: \"a2455bef-2bec-4e2e-9aed-f3734c47b8c9\") "
Sep 17 00:39:22.264480 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.264191    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f93433d-4c9e-40e1-8108-410cee61d3eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "0f93433d-4c9e-40e1-8108-410cee61d3eb" (UID: "0f93433d-4c9e-40e1-8108-410cee61d3eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:39:22.264480 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.264379    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-tmp-dir\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:22.264480 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.264384    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a1c787-acbc-4d1c-9657-b79b9db6de85-model-cache" (OuterVolumeSpecName: "model-cache") pod "97a1c787-acbc-4d1c-9657-b79b9db6de85" (UID: "97a1c787-acbc-4d1c-9657-b79b9db6de85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:39:22.264480 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.264396    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f93433d-4c9e-40e1-8108-410cee61d3eb-home\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:22.264480 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.264441    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a1c787-acbc-4d1c-9657-b79b9db6de85-tmp-dir\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:22.264480 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.264458    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f93433d-4c9e-40e1-8108-410cee61d3eb-model-cache\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:22.265080 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.264689    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a1c787-acbc-4d1c-9657-b79b9db6de85-home" (OuterVolumeSpecName: "home") pod "97a1c787-acbc-4d1c-9657-b79b9db6de85" (UID: "97a1c787-acbc-4d1c-9657-b79b9db6de85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:39:22.265388 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.265340    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-home" (OuterVolumeSpecName: "home") pod "a2455bef-2bec-4e2e-9aed-f3734c47b8c9" (UID: "a2455bef-2bec-4e2e-9aed-f3734c47b8c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:39:22.265541 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.265497    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2455bef-2bec-4e2e-9aed-f3734c47b8c9" (UID: "a2455bef-2bec-4e2e-9aed-f3734c47b8c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:39:22.265541 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.265510    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f93433d-4c9e-40e1-8108-410cee61d3eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0f93433d-4c9e-40e1-8108-410cee61d3eb" (UID: "0f93433d-4c9e-40e1-8108-410cee61d3eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:39:22.267236 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.267203    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a1c787-acbc-4d1c-9657-b79b9db6de85-dshm" (OuterVolumeSpecName: "dshm") pod "97a1c787-acbc-4d1c-9657-b79b9db6de85" (UID: "97a1c787-acbc-4d1c-9657-b79b9db6de85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:39:22.267236 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.267227    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f93433d-4c9e-40e1-8108-410cee61d3eb-dshm" (OuterVolumeSpecName: "dshm") pod "0f93433d-4c9e-40e1-8108-410cee61d3eb" (UID: "0f93433d-4c9e-40e1-8108-410cee61d3eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:39:22.267456 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.267273    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2455bef-2bec-4e2e-9aed-f3734c47b8c9" (UID: "a2455bef-2bec-4e2e-9aed-f3734c47b8c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:39:22.267456 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.267283    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a1c787-acbc-4d1c-9657-b79b9db6de85-kube-api-access-nqcxf" (OuterVolumeSpecName: "kube-api-access-nqcxf") pod "97a1c787-acbc-4d1c-9657-b79b9db6de85" (UID: "97a1c787-acbc-4d1c-9657-b79b9db6de85"). InnerVolumeSpecName "kube-api-access-nqcxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:39:22.267715 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.267688    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97a1c787-acbc-4d1c-9657-b79b9db6de85-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "97a1c787-acbc-4d1c-9657-b79b9db6de85" (UID: "97a1c787-acbc-4d1c-9657-b79b9db6de85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:39:22.267807 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.267785    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f93433d-4c9e-40e1-8108-410cee61d3eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f93433d-4c9e-40e1-8108-410cee61d3eb" (UID: "0f93433d-4c9e-40e1-8108-410cee61d3eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:39:22.267918 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.267884    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-kube-api-access-zm2r6" (OuterVolumeSpecName: "kube-api-access-zm2r6") pod "a2455bef-2bec-4e2e-9aed-f3734c47b8c9" (UID: "a2455bef-2bec-4e2e-9aed-f3734c47b8c9"). InnerVolumeSpecName "kube-api-access-zm2r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:39:22.268298 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.268273    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f93433d-4c9e-40e1-8108-410cee61d3eb-kube-api-access-9tc7f" (OuterVolumeSpecName: "kube-api-access-9tc7f") pod "0f93433d-4c9e-40e1-8108-410cee61d3eb" (UID: "0f93433d-4c9e-40e1-8108-410cee61d3eb"). InnerVolumeSpecName "kube-api-access-9tc7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:39:22.268764 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.268744    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-dshm" (OuterVolumeSpecName: "dshm") pod "a2455bef-2bec-4e2e-9aed-f3734c47b8c9" (UID: "a2455bef-2bec-4e2e-9aed-f3734c47b8c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:39:22.365347 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.365310    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a1c787-acbc-4d1c-9657-b79b9db6de85-model-cache\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:22.365347 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.365343    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f93433d-4c9e-40e1-8108-410cee61d3eb-dshm\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:22.365347 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.365353    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f93433d-4c9e-40e1-8108-410cee61d3eb-tls-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:22.365587 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.365362    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a1c787-acbc-4d1c-9657-b79b9db6de85-home\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:22.365587 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.365371    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm2r6\" (UniqueName: \"kubernetes.io/projected/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-kube-api-access-zm2r6\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:22.365587 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.365380    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tc7f\" (UniqueName: \"kubernetes.io/projected/0f93433d-4c9e-40e1-8108-410cee61d3eb-kube-api-access-9tc7f\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:22.365587 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.365388    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-dshm\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:22.365587 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.365401    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f93433d-4c9e-40e1-8108-410cee61d3eb-tmp-dir\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:22.365587 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.365425    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-model-cache\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:22.365587 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.365432    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-tls-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:22.365587 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.365440    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqcxf\" (UniqueName: \"kubernetes.io/projected/97a1c787-acbc-4d1c-9657-b79b9db6de85-kube-api-access-nqcxf\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:22.365587 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.365447    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a1c787-acbc-4d1c-9657-b79b9db6de85-dshm\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:22.365587 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.365454    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2455bef-2bec-4e2e-9aed-f3734c47b8c9-home\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:22.365587 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.365461    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a1c787-acbc-4d1c-9657-b79b9db6de85-tls-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:22.731554 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.731518    2577 generic.go:356] "Generic (PLEG): container finished" podID="0f93433d-4c9e-40e1-8108-410cee61d3eb" containerID="9b7a108b97fe2af9a0871cb563e76c5ab1ffb25ec64a116b3330bd29d559e4a5" exitCode=0
Sep 17 00:39:22.731755 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.731560    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"
Sep 17 00:39:22.731755 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.731599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r" event={"ID":"0f93433d-4c9e-40e1-8108-410cee61d3eb","Type":"ContainerDied","Data":"9b7a108b97fe2af9a0871cb563e76c5ab1ffb25ec64a116b3330bd29d559e4a5"}
Sep 17 00:39:22.731755 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.731639    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r" event={"ID":"0f93433d-4c9e-40e1-8108-410cee61d3eb","Type":"ContainerDied","Data":"7403ffbf1dd75df6db6d3275dfbec2eb7848ffcc19fb02f55f123d3900025ed1"}
Sep 17 00:39:22.731755 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.731657    2577 scope.go:117] "RemoveContainer" containerID="9b7a108b97fe2af9a0871cb563e76c5ab1ffb25ec64a116b3330bd29d559e4a5"
Sep 17 00:39:22.733052 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.733021    2577 generic.go:356] "Generic (PLEG): container finished" podID="a2455bef-2bec-4e2e-9aed-f3734c47b8c9" containerID="d0e41ead288ee83216691e320f8af73ac8812d00a6452faf59845111921d8595" exitCode=0
Sep 17 00:39:22.733243 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.733059    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"
Sep 17 00:39:22.733243 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.733091    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf" event={"ID":"a2455bef-2bec-4e2e-9aed-f3734c47b8c9","Type":"ContainerDied","Data":"d0e41ead288ee83216691e320f8af73ac8812d00a6452faf59845111921d8595"}
Sep 17 00:39:22.733243 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.733121    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf" event={"ID":"a2455bef-2bec-4e2e-9aed-f3734c47b8c9","Type":"ContainerDied","Data":"b3de551273098f31fcba10820fe4c5c1114393bbbe5fb3e0652558dc91975b6a"}
Sep 17 00:39:22.734793 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.734769    2577 generic.go:356] "Generic (PLEG): container finished" podID="97a1c787-acbc-4d1c-9657-b79b9db6de85" containerID="7baf95da4bc3e9f9df760a9b8d92891f81a2eb5aff8267189512ba7ac7720bf0" exitCode=0
Sep 17 00:39:22.734890 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.734827    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8" event={"ID":"97a1c787-acbc-4d1c-9657-b79b9db6de85","Type":"ContainerDied","Data":"7baf95da4bc3e9f9df760a9b8d92891f81a2eb5aff8267189512ba7ac7720bf0"}
Sep 17 00:39:22.734890 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.734858    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8" event={"ID":"97a1c787-acbc-4d1c-9657-b79b9db6de85","Type":"ContainerDied","Data":"c2c6929a348faeee086af8ed7393e61906864ee0ee2ed1abe10029199eab07dc"}
Sep 17 00:39:22.734985 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.734907    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"
Sep 17 00:39:22.742604 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.742587    2577 scope.go:117] "RemoveContainer" containerID="9b7a108b97fe2af9a0871cb563e76c5ab1ffb25ec64a116b3330bd29d559e4a5"
Sep 17 00:39:22.743014 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:39:22.742962    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b7a108b97fe2af9a0871cb563e76c5ab1ffb25ec64a116b3330bd29d559e4a5\": container with ID starting with 9b7a108b97fe2af9a0871cb563e76c5ab1ffb25ec64a116b3330bd29d559e4a5 not found: ID does not exist" containerID="9b7a108b97fe2af9a0871cb563e76c5ab1ffb25ec64a116b3330bd29d559e4a5"
Sep 17 00:39:22.743014 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.742997    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7a108b97fe2af9a0871cb563e76c5ab1ffb25ec64a116b3330bd29d559e4a5"} err="failed to get container status \"9b7a108b97fe2af9a0871cb563e76c5ab1ffb25ec64a116b3330bd29d559e4a5\": rpc error: code = NotFound desc = could not find container \"9b7a108b97fe2af9a0871cb563e76c5ab1ffb25ec64a116b3330bd29d559e4a5\": container with ID starting with 9b7a108b97fe2af9a0871cb563e76c5ab1ffb25ec64a116b3330bd29d559e4a5 not found: ID does not exist"
Sep 17 00:39:22.743128 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.743019    2577 scope.go:117] "RemoveContainer" containerID="d0e41ead288ee83216691e320f8af73ac8812d00a6452faf59845111921d8595"
Sep 17 00:39:22.752204 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.752185    2577 scope.go:117] "RemoveContainer" containerID="d0e41ead288ee83216691e320f8af73ac8812d00a6452faf59845111921d8595"
Sep 17 00:39:22.752507 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:39:22.752484    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0e41ead288ee83216691e320f8af73ac8812d00a6452faf59845111921d8595\": container with ID starting with d0e41ead288ee83216691e320f8af73ac8812d00a6452faf59845111921d8595 not found: ID does not exist" containerID="d0e41ead288ee83216691e320f8af73ac8812d00a6452faf59845111921d8595"
Sep 17 00:39:22.752568 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.752517    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0e41ead288ee83216691e320f8af73ac8812d00a6452faf59845111921d8595"} err="failed to get container status \"d0e41ead288ee83216691e320f8af73ac8812d00a6452faf59845111921d8595\": rpc error: code = NotFound desc = could not find container \"d0e41ead288ee83216691e320f8af73ac8812d00a6452faf59845111921d8595\": container with ID starting with d0e41ead288ee83216691e320f8af73ac8812d00a6452faf59845111921d8595 not found: ID does not exist"
Sep 17 00:39:22.752568 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.752542    2577 scope.go:117] "RemoveContainer" containerID="7baf95da4bc3e9f9df760a9b8d92891f81a2eb5aff8267189512ba7ac7720bf0"
Sep 17 00:39:22.758726 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.758706    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"]
Sep 17 00:39:22.762574 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.762555    2577 scope.go:117] "RemoveContainer" containerID="7baf95da4bc3e9f9df760a9b8d92891f81a2eb5aff8267189512ba7ac7720bf0"
Sep 17 00:39:22.762792 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.762774    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-cfe049/premium-w0-cfe049-kserve-85dbc8fc77-wkw4r"]
Sep 17 00:39:22.762843 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:39:22.762829    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7baf95da4bc3e9f9df760a9b8d92891f81a2eb5aff8267189512ba7ac7720bf0\": container with ID starting with 7baf95da4bc3e9f9df760a9b8d92891f81a2eb5aff8267189512ba7ac7720bf0 not found: ID does not exist" containerID="7baf95da4bc3e9f9df760a9b8d92891f81a2eb5aff8267189512ba7ac7720bf0"
Sep 17 00:39:22.762889 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.762847    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7baf95da4bc3e9f9df760a9b8d92891f81a2eb5aff8267189512ba7ac7720bf0"} err="failed to get container status \"7baf95da4bc3e9f9df760a9b8d92891f81a2eb5aff8267189512ba7ac7720bf0\": rpc error: code = NotFound desc = could not find container \"7baf95da4bc3e9f9df760a9b8d92891f81a2eb5aff8267189512ba7ac7720bf0\": container with ID starting with 7baf95da4bc3e9f9df760a9b8d92891f81a2eb5aff8267189512ba7ac7720bf0 not found: ID does not exist"
Sep 17 00:39:22.770451 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.770406    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"]
Sep 17 00:39:22.773949 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.773927    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-cfe049/distinct-w0-cfe049-kserve-5db7d67668-hpvgf"]
Sep 17 00:39:22.782997 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.782978    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"]
Sep 17 00:39:22.785042 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:22.785021    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-cfe049/simulator-w0-cfe049-kserve-5c5797bdfb-nqcb8"]
Sep 17 00:39:23.608179 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:23.608143    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f93433d-4c9e-40e1-8108-410cee61d3eb" path="/var/lib/kubelet/pods/0f93433d-4c9e-40e1-8108-410cee61d3eb/volumes"
Sep 17 00:39:23.608743 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:23.608559    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97a1c787-acbc-4d1c-9657-b79b9db6de85" path="/var/lib/kubelet/pods/97a1c787-acbc-4d1c-9657-b79b9db6de85/volumes"
Sep 17 00:39:23.608926 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:23.608912    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2455bef-2bec-4e2e-9aed-f3734c47b8c9" path="/var/lib/kubelet/pods/a2455bef-2bec-4e2e-9aed-f3734c47b8c9/volumes"
Sep 17 00:39:37.198330 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:37.198295    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r"]
Sep 17 00:39:37.198808 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:37.198600    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r" podUID="b86d40c8-5566-45fe-a649-faf8a7231e95" containerName="maas-api" containerID="cri-o://3f057d2d7a6002c5b660cc69974c6cc683015904a1475331fea850e26d661c44" gracePeriod=30
Sep 17 00:39:37.443317 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:37.443292    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r"
Sep 17 00:39:37.504772 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:37.504685    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86d40c8-5566-45fe-a649-faf8a7231e95-metrics-certs\") pod \"b86d40c8-5566-45fe-a649-faf8a7231e95\" (UID: \"b86d40c8-5566-45fe-a649-faf8a7231e95\") "
Sep 17 00:39:37.504920 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:37.504793    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b86d40c8-5566-45fe-a649-faf8a7231e95-openshift-service-ca\") pod \"b86d40c8-5566-45fe-a649-faf8a7231e95\" (UID: \"b86d40c8-5566-45fe-a649-faf8a7231e95\") "
Sep 17 00:39:37.504920 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:37.504848    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b86d40c8-5566-45fe-a649-faf8a7231e95-maas-api-tls\") pod \"b86d40c8-5566-45fe-a649-faf8a7231e95\" (UID: \"b86d40c8-5566-45fe-a649-faf8a7231e95\") "
Sep 17 00:39:37.504920 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:37.504899    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpc8g\" (UniqueName: \"kubernetes.io/projected/b86d40c8-5566-45fe-a649-faf8a7231e95-kube-api-access-dpc8g\") pod \"b86d40c8-5566-45fe-a649-faf8a7231e95\" (UID: \"b86d40c8-5566-45fe-a649-faf8a7231e95\") "
Sep 17 00:39:37.505158 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:37.505131    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86d40c8-5566-45fe-a649-faf8a7231e95-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b86d40c8-5566-45fe-a649-faf8a7231e95" (UID: "b86d40c8-5566-45fe-a649-faf8a7231e95"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:39:37.506995 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:37.506956    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86d40c8-5566-45fe-a649-faf8a7231e95-kube-api-access-dpc8g" (OuterVolumeSpecName: "kube-api-access-dpc8g") pod "b86d40c8-5566-45fe-a649-faf8a7231e95" (UID: "b86d40c8-5566-45fe-a649-faf8a7231e95"). InnerVolumeSpecName "kube-api-access-dpc8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:39:37.506995 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:37.506954    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b86d40c8-5566-45fe-a649-faf8a7231e95-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b86d40c8-5566-45fe-a649-faf8a7231e95" (UID: "b86d40c8-5566-45fe-a649-faf8a7231e95"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:39:37.507154 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:37.506995    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b86d40c8-5566-45fe-a649-faf8a7231e95-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b86d40c8-5566-45fe-a649-faf8a7231e95" (UID: "b86d40c8-5566-45fe-a649-faf8a7231e95"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:39:37.606072 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:37.606035    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b86d40c8-5566-45fe-a649-faf8a7231e95-maas-api-tls\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:37.606072 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:37.606063    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpc8g\" (UniqueName: \"kubernetes.io/projected/b86d40c8-5566-45fe-a649-faf8a7231e95-kube-api-access-dpc8g\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:37.606072 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:37.606075    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86d40c8-5566-45fe-a649-faf8a7231e95-metrics-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:37.606317 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:37.606083    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b86d40c8-5566-45fe-a649-faf8a7231e95-openshift-service-ca\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:39:37.823324 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:37.823234    2577 generic.go:356] "Generic (PLEG): container finished" podID="b86d40c8-5566-45fe-a649-faf8a7231e95" containerID="3f057d2d7a6002c5b660cc69974c6cc683015904a1475331fea850e26d661c44" exitCode=0
Sep 17 00:39:37.823324 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:37.823279    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r" event={"ID":"b86d40c8-5566-45fe-a649-faf8a7231e95","Type":"ContainerDied","Data":"3f057d2d7a6002c5b660cc69974c6cc683015904a1475331fea850e26d661c44"}
Sep 17 00:39:37.823324 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:37.823289    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r"
Sep 17 00:39:37.823611 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:37.823334    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r" event={"ID":"b86d40c8-5566-45fe-a649-faf8a7231e95","Type":"ContainerDied","Data":"616c12977c87c4a6fa053aadbe172d7f594293b743c92cbf06cce4dca554de31"}
Sep 17 00:39:37.823611 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:37.823366    2577 scope.go:117] "RemoveContainer" containerID="3f057d2d7a6002c5b660cc69974c6cc683015904a1475331fea850e26d661c44"
Sep 17 00:39:37.832511 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:37.832452    2577 scope.go:117] "RemoveContainer" containerID="3f057d2d7a6002c5b660cc69974c6cc683015904a1475331fea850e26d661c44"
Sep 17 00:39:37.832756 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:39:37.832736    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f057d2d7a6002c5b660cc69974c6cc683015904a1475331fea850e26d661c44\": container with ID starting with 3f057d2d7a6002c5b660cc69974c6cc683015904a1475331fea850e26d661c44 not found: ID does not exist" containerID="3f057d2d7a6002c5b660cc69974c6cc683015904a1475331fea850e26d661c44"
Sep 17 00:39:37.832799 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:37.832768    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f057d2d7a6002c5b660cc69974c6cc683015904a1475331fea850e26d661c44"} err="failed to get container status \"3f057d2d7a6002c5b660cc69974c6cc683015904a1475331fea850e26d661c44\": rpc error: code = NotFound desc = could not find container \"3f057d2d7a6002c5b660cc69974c6cc683015904a1475331fea850e26d661c44\": container with ID starting with 3f057d2d7a6002c5b660cc69974c6cc683015904a1475331fea850e26d661c44 not found: ID does not exist"
Sep 17 00:39:37.839601 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:37.839572    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r"]
Sep 17 00:39:37.841703 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:37.841678    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-48e479-654c7f97f7-27s6r"]
Sep 17 00:39:39.608176 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:39.608142    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b86d40c8-5566-45fe-a649-faf8a7231e95" path="/var/lib/kubelet/pods/b86d40c8-5566-45fe-a649-faf8a7231e95/volumes"
Sep 17 00:39:46.201883 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:46.201847    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:39:47.704107 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:47.704079    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/ovn-acl-logging/0.log"
Sep 17 00:39:47.713221 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:47.713201    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/ovn-acl-logging/0.log"
Sep 17 00:39:47.842729 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:47.842692    2577 scope.go:117] "RemoveContainer" containerID="f5411c6aa3200f77781234b68db947dbb432d435aca80b83890207c7b8c107ca"
Sep 17 00:39:56.431685 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:56.431654    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"]
Sep 17 00:39:56.435811 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:56.435780    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"]
Sep 17 00:39:56.452282 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:39:56.452236    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"]
Sep 17 00:40:03.407093 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:03.407049    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:40:07.411000 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:07.410962    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj"]
Sep 17 00:40:07.411552 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:07.411255    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj" podUID="95df6c2c-6af3-44c2-bed7-399abf2d29e9" containerName="maas-api" containerID="cri-o://b9a757ff59e15c83968c1ca41741b581d3258ed8e63844f9910b982abf543577" gracePeriod=30
Sep 17 00:40:07.857203 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:07.857176    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj"
Sep 17 00:40:07.879558 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:07.879430    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95df6c2c-6af3-44c2-bed7-399abf2d29e9-metrics-certs\") pod \"95df6c2c-6af3-44c2-bed7-399abf2d29e9\" (UID: \"95df6c2c-6af3-44c2-bed7-399abf2d29e9\") "
Sep 17 00:40:07.879558 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:07.879488    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95df6c2c-6af3-44c2-bed7-399abf2d29e9-maas-api-tls\") pod \"95df6c2c-6af3-44c2-bed7-399abf2d29e9\" (UID: \"95df6c2c-6af3-44c2-bed7-399abf2d29e9\") "
Sep 17 00:40:07.879772 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:07.879577    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljrgt\" (UniqueName: \"kubernetes.io/projected/95df6c2c-6af3-44c2-bed7-399abf2d29e9-kube-api-access-ljrgt\") pod \"95df6c2c-6af3-44c2-bed7-399abf2d29e9\" (UID: \"95df6c2c-6af3-44c2-bed7-399abf2d29e9\") "
Sep 17 00:40:07.879772 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:07.879653    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95df6c2c-6af3-44c2-bed7-399abf2d29e9-openshift-service-ca\") pod \"95df6c2c-6af3-44c2-bed7-399abf2d29e9\" (UID: \"95df6c2c-6af3-44c2-bed7-399abf2d29e9\") "
Sep 17 00:40:07.880145 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:07.880105    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95df6c2c-6af3-44c2-bed7-399abf2d29e9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "95df6c2c-6af3-44c2-bed7-399abf2d29e9" (UID: "95df6c2c-6af3-44c2-bed7-399abf2d29e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:40:07.881742 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:07.881716    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95df6c2c-6af3-44c2-bed7-399abf2d29e9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "95df6c2c-6af3-44c2-bed7-399abf2d29e9" (UID: "95df6c2c-6af3-44c2-bed7-399abf2d29e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:40:07.882097 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:07.882076    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95df6c2c-6af3-44c2-bed7-399abf2d29e9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "95df6c2c-6af3-44c2-bed7-399abf2d29e9" (UID: "95df6c2c-6af3-44c2-bed7-399abf2d29e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:40:07.882172 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:07.882114    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95df6c2c-6af3-44c2-bed7-399abf2d29e9-kube-api-access-ljrgt" (OuterVolumeSpecName: "kube-api-access-ljrgt") pod "95df6c2c-6af3-44c2-bed7-399abf2d29e9" (UID: "95df6c2c-6af3-44c2-bed7-399abf2d29e9"). InnerVolumeSpecName "kube-api-access-ljrgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:40:07.981390 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:07.981312    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljrgt\" (UniqueName: \"kubernetes.io/projected/95df6c2c-6af3-44c2-bed7-399abf2d29e9-kube-api-access-ljrgt\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:40:07.981390 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:07.981340    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95df6c2c-6af3-44c2-bed7-399abf2d29e9-openshift-service-ca\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:40:07.981390 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:07.981355    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95df6c2c-6af3-44c2-bed7-399abf2d29e9-metrics-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:40:07.981390 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:07.981364    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95df6c2c-6af3-44c2-bed7-399abf2d29e9-maas-api-tls\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:40:07.997007 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:07.996975    2577 generic.go:356] "Generic (PLEG): container finished" podID="95df6c2c-6af3-44c2-bed7-399abf2d29e9" containerID="b9a757ff59e15c83968c1ca41741b581d3258ed8e63844f9910b982abf543577" exitCode=0
Sep 17 00:40:07.997132 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:07.997008    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj" event={"ID":"95df6c2c-6af3-44c2-bed7-399abf2d29e9","Type":"ContainerDied","Data":"b9a757ff59e15c83968c1ca41741b581d3258ed8e63844f9910b982abf543577"}
Sep 17 00:40:07.997132 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:07.997063    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj" event={"ID":"95df6c2c-6af3-44c2-bed7-399abf2d29e9","Type":"ContainerDied","Data":"5a060d5a1ddc0d71d72a7dc46ba47e48a4d4d4526d7ca3809d51b42fbf336ba1"}
Sep 17 00:40:07.997132 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:07.997086    2577 scope.go:117] "RemoveContainer" containerID="b9a757ff59e15c83968c1ca41741b581d3258ed8e63844f9910b982abf543577"
Sep 17 00:40:07.997132 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:07.997023    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj"
Sep 17 00:40:08.007071 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:08.007051    2577 scope.go:117] "RemoveContainer" containerID="b9a757ff59e15c83968c1ca41741b581d3258ed8e63844f9910b982abf543577"
Sep 17 00:40:08.007344 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:40:08.007323    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a757ff59e15c83968c1ca41741b581d3258ed8e63844f9910b982abf543577\": container with ID starting with b9a757ff59e15c83968c1ca41741b581d3258ed8e63844f9910b982abf543577 not found: ID does not exist" containerID="b9a757ff59e15c83968c1ca41741b581d3258ed8e63844f9910b982abf543577"
Sep 17 00:40:08.007389 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:08.007353    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a757ff59e15c83968c1ca41741b581d3258ed8e63844f9910b982abf543577"} err="failed to get container status \"b9a757ff59e15c83968c1ca41741b581d3258ed8e63844f9910b982abf543577\": rpc error: code = NotFound desc = could not find container \"b9a757ff59e15c83968c1ca41741b581d3258ed8e63844f9910b982abf543577\": container with ID starting with b9a757ff59e15c83968c1ca41741b581d3258ed8e63844f9910b982abf543577 not found: ID does not exist"
Sep 17 00:40:08.019530 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:08.019505    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj"]
Sep 17 00:40:08.022529 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:08.022507    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-2faea3-7bc46dbbb5-5t2tj"]
Sep 17 00:40:09.607899 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:09.607864    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95df6c2c-6af3-44c2-bed7-399abf2d29e9" path="/var/lib/kubelet/pods/95df6c2c-6af3-44c2-bed7-399abf2d29e9/volumes"
Sep 17 00:40:11.454806 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.454740    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6" podUID="1886bc33-36ad-492e-a0c5-a2a317988c34" containerName="main" containerID="cri-o://aad81999c131fffef22c07c020895981989aa4c6fce7e8d5507eceef729a33ce" gracePeriod=45
Sep 17 00:40:11.456803 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.456752    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8" podUID="63be31f1-7be5-4cc6-a614-271adaa92912" containerName="main" containerID="cri-o://3349f89d6ee5a70aba77cf304176b23e308f5462b2306735ec317d8ce18b10d6" gracePeriod=45
Sep 17 00:40:11.471993 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.471964    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg" podUID="e533b075-a947-45b0-bbbf-1ac95ade2554" containerName="main" containerID="cri-o://ab7536cd8f3af8d7bfb7c13226d84cf3fd71979c255a4e23eae07e38f8de0cd6" gracePeriod=45
Sep 17 00:40:11.761921 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.761890    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:40:11.765263 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.765235    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:40:11.768629 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.768605    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:40:11.815333 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.815301    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1886bc33-36ad-492e-a0c5-a2a317988c34-tmp-dir\") pod \"1886bc33-36ad-492e-a0c5-a2a317988c34\" (UID: \"1886bc33-36ad-492e-a0c5-a2a317988c34\") "
Sep 17 00:40:11.815525 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.815357    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e533b075-a947-45b0-bbbf-1ac95ade2554-home\") pod \"e533b075-a947-45b0-bbbf-1ac95ade2554\" (UID: \"e533b075-a947-45b0-bbbf-1ac95ade2554\") "
Sep 17 00:40:11.815525 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.815387    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63be31f1-7be5-4cc6-a614-271adaa92912-model-cache\") pod \"63be31f1-7be5-4cc6-a614-271adaa92912\" (UID: \"63be31f1-7be5-4cc6-a614-271adaa92912\") "
Sep 17 00:40:11.815525 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.815435    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e533b075-a947-45b0-bbbf-1ac95ade2554-tls-certs\") pod \"e533b075-a947-45b0-bbbf-1ac95ade2554\" (UID: \"e533b075-a947-45b0-bbbf-1ac95ade2554\") "
Sep 17 00:40:11.815525 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.815464    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63be31f1-7be5-4cc6-a614-271adaa92912-home\") pod \"63be31f1-7be5-4cc6-a614-271adaa92912\" (UID: \"63be31f1-7be5-4cc6-a614-271adaa92912\") "
Sep 17 00:40:11.815525 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.815490    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e533b075-a947-45b0-bbbf-1ac95ade2554-model-cache\") pod \"e533b075-a947-45b0-bbbf-1ac95ade2554\" (UID: \"e533b075-a947-45b0-bbbf-1ac95ade2554\") "
Sep 17 00:40:11.815525 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.815527    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1886bc33-36ad-492e-a0c5-a2a317988c34-home\") pod \"1886bc33-36ad-492e-a0c5-a2a317988c34\" (UID: \"1886bc33-36ad-492e-a0c5-a2a317988c34\") "
Sep 17 00:40:11.815856 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.815555    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btbjg\" (UniqueName: \"kubernetes.io/projected/1886bc33-36ad-492e-a0c5-a2a317988c34-kube-api-access-btbjg\") pod \"1886bc33-36ad-492e-a0c5-a2a317988c34\" (UID: \"1886bc33-36ad-492e-a0c5-a2a317988c34\") "
Sep 17 00:40:11.815856 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.815578    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e533b075-a947-45b0-bbbf-1ac95ade2554-tmp-dir\") pod \"e533b075-a947-45b0-bbbf-1ac95ade2554\" (UID: \"e533b075-a947-45b0-bbbf-1ac95ade2554\") "
Sep 17 00:40:11.815856 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.815625    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63be31f1-7be5-4cc6-a614-271adaa92912-dshm\") pod \"63be31f1-7be5-4cc6-a614-271adaa92912\" (UID: \"63be31f1-7be5-4cc6-a614-271adaa92912\") "
Sep 17 00:40:11.815856 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.815654    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1886bc33-36ad-492e-a0c5-a2a317988c34-model-cache\") pod \"1886bc33-36ad-492e-a0c5-a2a317988c34\" (UID: \"1886bc33-36ad-492e-a0c5-a2a317988c34\") "
Sep 17 00:40:11.816068 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.815620    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1886bc33-36ad-492e-a0c5-a2a317988c34-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1886bc33-36ad-492e-a0c5-a2a317988c34" (UID: "1886bc33-36ad-492e-a0c5-a2a317988c34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:40:11.816068 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.815963    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63be31f1-7be5-4cc6-a614-271adaa92912-model-cache" (OuterVolumeSpecName: "model-cache") pod "63be31f1-7be5-4cc6-a614-271adaa92912" (UID: "63be31f1-7be5-4cc6-a614-271adaa92912"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:40:11.816068 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.815975    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e533b075-a947-45b0-bbbf-1ac95ade2554-model-cache" (OuterVolumeSpecName: "model-cache") pod "e533b075-a947-45b0-bbbf-1ac95ade2554" (UID: "e533b075-a947-45b0-bbbf-1ac95ade2554"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:40:11.816236 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.816214    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e533b075-a947-45b0-bbbf-1ac95ade2554-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e533b075-a947-45b0-bbbf-1ac95ade2554" (UID: "e533b075-a947-45b0-bbbf-1ac95ade2554"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:40:11.816288 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.816236    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1886bc33-36ad-492e-a0c5-a2a317988c34-home" (OuterVolumeSpecName: "home") pod "1886bc33-36ad-492e-a0c5-a2a317988c34" (UID: "1886bc33-36ad-492e-a0c5-a2a317988c34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:40:11.816351 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.816279    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1886bc33-36ad-492e-a0c5-a2a317988c34-model-cache" (OuterVolumeSpecName: "model-cache") pod "1886bc33-36ad-492e-a0c5-a2a317988c34" (UID: "1886bc33-36ad-492e-a0c5-a2a317988c34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:40:11.816351 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.816277    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63be31f1-7be5-4cc6-a614-271adaa92912-home" (OuterVolumeSpecName: "home") pod "63be31f1-7be5-4cc6-a614-271adaa92912" (UID: "63be31f1-7be5-4cc6-a614-271adaa92912"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:40:11.816471 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.816367    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1886bc33-36ad-492e-a0c5-a2a317988c34-dshm\") pod \"1886bc33-36ad-492e-a0c5-a2a317988c34\" (UID: \"1886bc33-36ad-492e-a0c5-a2a317988c34\") "
Sep 17 00:40:11.816471 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.816409    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c4r8\" (UniqueName: \"kubernetes.io/projected/e533b075-a947-45b0-bbbf-1ac95ade2554-kube-api-access-4c4r8\") pod \"e533b075-a947-45b0-bbbf-1ac95ade2554\" (UID: \"e533b075-a947-45b0-bbbf-1ac95ade2554\") "
Sep 17 00:40:11.816586 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.816468    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e533b075-a947-45b0-bbbf-1ac95ade2554-dshm\") pod \"e533b075-a947-45b0-bbbf-1ac95ade2554\" (UID: \"e533b075-a947-45b0-bbbf-1ac95ade2554\") "
Sep 17 00:40:11.816586 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.816493    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63be31f1-7be5-4cc6-a614-271adaa92912-tmp-dir\") pod \"63be31f1-7be5-4cc6-a614-271adaa92912\" (UID: \"63be31f1-7be5-4cc6-a614-271adaa92912\") "
Sep 17 00:40:11.816707 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.816679    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e533b075-a947-45b0-bbbf-1ac95ade2554-home" (OuterVolumeSpecName: "home") pod "e533b075-a947-45b0-bbbf-1ac95ade2554" (UID: "e533b075-a947-45b0-bbbf-1ac95ade2554"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:40:11.816707 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.816697    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63be31f1-7be5-4cc6-a614-271adaa92912-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "63be31f1-7be5-4cc6-a614-271adaa92912" (UID: "63be31f1-7be5-4cc6-a614-271adaa92912"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:40:11.816818 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.816737    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63be31f1-7be5-4cc6-a614-271adaa92912-tls-certs\") pod \"63be31f1-7be5-4cc6-a614-271adaa92912\" (UID: \"63be31f1-7be5-4cc6-a614-271adaa92912\") "
Sep 17 00:40:11.816818 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.816773    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1886bc33-36ad-492e-a0c5-a2a317988c34-tls-certs\") pod \"1886bc33-36ad-492e-a0c5-a2a317988c34\" (UID: \"1886bc33-36ad-492e-a0c5-a2a317988c34\") "
Sep 17 00:40:11.816921 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.816828    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mfs9\" (UniqueName: \"kubernetes.io/projected/63be31f1-7be5-4cc6-a614-271adaa92912-kube-api-access-6mfs9\") pod \"63be31f1-7be5-4cc6-a614-271adaa92912\" (UID: \"63be31f1-7be5-4cc6-a614-271adaa92912\") "
Sep 17 00:40:11.817222 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.817116    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e533b075-a947-45b0-bbbf-1ac95ade2554-home\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:40:11.817222 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.817140    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63be31f1-7be5-4cc6-a614-271adaa92912-model-cache\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:40:11.817222 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.817155    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63be31f1-7be5-4cc6-a614-271adaa92912-home\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:40:11.817222 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.817169    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e533b075-a947-45b0-bbbf-1ac95ade2554-model-cache\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:40:11.817222 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.817181    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1886bc33-36ad-492e-a0c5-a2a317988c34-home\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:40:11.817222 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.817216    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e533b075-a947-45b0-bbbf-1ac95ade2554-tmp-dir\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:40:11.817605 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.817229    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1886bc33-36ad-492e-a0c5-a2a317988c34-model-cache\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:40:11.817605 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.817240    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63be31f1-7be5-4cc6-a614-271adaa92912-tmp-dir\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:40:11.817605 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.817250    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1886bc33-36ad-492e-a0c5-a2a317988c34-tmp-dir\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:40:11.819335 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.819302    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1886bc33-36ad-492e-a0c5-a2a317988c34-kube-api-access-btbjg" (OuterVolumeSpecName: "kube-api-access-btbjg") pod "1886bc33-36ad-492e-a0c5-a2a317988c34" (UID: "1886bc33-36ad-492e-a0c5-a2a317988c34"). InnerVolumeSpecName "kube-api-access-btbjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:40:11.819493 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.819346    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1886bc33-36ad-492e-a0c5-a2a317988c34-dshm" (OuterVolumeSpecName: "dshm") pod "1886bc33-36ad-492e-a0c5-a2a317988c34" (UID: "1886bc33-36ad-492e-a0c5-a2a317988c34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:40:11.819493 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.819394    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e533b075-a947-45b0-bbbf-1ac95ade2554-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e533b075-a947-45b0-bbbf-1ac95ade2554" (UID: "e533b075-a947-45b0-bbbf-1ac95ade2554"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:40:11.820160 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.820138    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1886bc33-36ad-492e-a0c5-a2a317988c34-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1886bc33-36ad-492e-a0c5-a2a317988c34" (UID: "1886bc33-36ad-492e-a0c5-a2a317988c34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:40:11.820542 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.820523    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e533b075-a947-45b0-bbbf-1ac95ade2554-dshm" (OuterVolumeSpecName: "dshm") pod "e533b075-a947-45b0-bbbf-1ac95ade2554" (UID: "e533b075-a947-45b0-bbbf-1ac95ade2554"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:40:11.820640 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.820582    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63be31f1-7be5-4cc6-a614-271adaa92912-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "63be31f1-7be5-4cc6-a614-271adaa92912" (UID: "63be31f1-7be5-4cc6-a614-271adaa92912"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:40:11.820640 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.820632    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63be31f1-7be5-4cc6-a614-271adaa92912-dshm" (OuterVolumeSpecName: "dshm") pod "63be31f1-7be5-4cc6-a614-271adaa92912" (UID: "63be31f1-7be5-4cc6-a614-271adaa92912"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:40:11.821251 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.821228    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e533b075-a947-45b0-bbbf-1ac95ade2554-kube-api-access-4c4r8" (OuterVolumeSpecName: "kube-api-access-4c4r8") pod "e533b075-a947-45b0-bbbf-1ac95ade2554" (UID: "e533b075-a947-45b0-bbbf-1ac95ade2554"). InnerVolumeSpecName "kube-api-access-4c4r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:40:11.821484 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.821458    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63be31f1-7be5-4cc6-a614-271adaa92912-kube-api-access-6mfs9" (OuterVolumeSpecName: "kube-api-access-6mfs9") pod "63be31f1-7be5-4cc6-a614-271adaa92912" (UID: "63be31f1-7be5-4cc6-a614-271adaa92912"). InnerVolumeSpecName "kube-api-access-6mfs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:40:11.918408 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.918371    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mfs9\" (UniqueName: \"kubernetes.io/projected/63be31f1-7be5-4cc6-a614-271adaa92912-kube-api-access-6mfs9\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:40:11.918408 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.918402    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e533b075-a947-45b0-bbbf-1ac95ade2554-tls-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:40:11.918408 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.918436    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btbjg\" (UniqueName: \"kubernetes.io/projected/1886bc33-36ad-492e-a0c5-a2a317988c34-kube-api-access-btbjg\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:40:11.918671 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.918448    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63be31f1-7be5-4cc6-a614-271adaa92912-dshm\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:40:11.918671 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.918461    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1886bc33-36ad-492e-a0c5-a2a317988c34-dshm\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:40:11.918671 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.918472    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c4r8\" (UniqueName: \"kubernetes.io/projected/e533b075-a947-45b0-bbbf-1ac95ade2554-kube-api-access-4c4r8\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:40:11.918671 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.918482    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e533b075-a947-45b0-bbbf-1ac95ade2554-dshm\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:40:11.918671 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.918495    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63be31f1-7be5-4cc6-a614-271adaa92912-tls-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:40:11.918671 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:11.918506    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1886bc33-36ad-492e-a0c5-a2a317988c34-tls-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:40:12.021178 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.021084    2577 generic.go:356] "Generic (PLEG): container finished" podID="1886bc33-36ad-492e-a0c5-a2a317988c34" containerID="aad81999c131fffef22c07c020895981989aa4c6fce7e8d5507eceef729a33ce" exitCode=0
Sep 17 00:40:12.021178 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.021134    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"
Sep 17 00:40:12.021178 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.021166    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6" event={"ID":"1886bc33-36ad-492e-a0c5-a2a317988c34","Type":"ContainerDied","Data":"aad81999c131fffef22c07c020895981989aa4c6fce7e8d5507eceef729a33ce"}
Sep 17 00:40:12.021539 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.021219    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6" event={"ID":"1886bc33-36ad-492e-a0c5-a2a317988c34","Type":"ContainerDied","Data":"cf9105b273ed5fb16788608a94542a76dd2823d6a8833af90e0f7fa93db9c118"}
Sep 17 00:40:12.021539 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.021241    2577 scope.go:117] "RemoveContainer" containerID="aad81999c131fffef22c07c020895981989aa4c6fce7e8d5507eceef729a33ce"
Sep 17 00:40:12.022940 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.022916    2577 generic.go:356] "Generic (PLEG): container finished" podID="63be31f1-7be5-4cc6-a614-271adaa92912" containerID="3349f89d6ee5a70aba77cf304176b23e308f5462b2306735ec317d8ce18b10d6" exitCode=0
Sep 17 00:40:12.023055 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.022957    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"
Sep 17 00:40:12.023055 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.022994    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8" event={"ID":"63be31f1-7be5-4cc6-a614-271adaa92912","Type":"ContainerDied","Data":"3349f89d6ee5a70aba77cf304176b23e308f5462b2306735ec317d8ce18b10d6"}
Sep 17 00:40:12.023055 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.023034    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8" event={"ID":"63be31f1-7be5-4cc6-a614-271adaa92912","Type":"ContainerDied","Data":"08b668680f370a143cbe4dcf91c8281dc19ecc95e26656294aed8594e22c2ea6"}
Sep 17 00:40:12.025263 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.024579    2577 generic.go:356] "Generic (PLEG): container finished" podID="e533b075-a947-45b0-bbbf-1ac95ade2554" containerID="ab7536cd8f3af8d7bfb7c13226d84cf3fd71979c255a4e23eae07e38f8de0cd6" exitCode=0
Sep 17 00:40:12.025263 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.024609    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg" event={"ID":"e533b075-a947-45b0-bbbf-1ac95ade2554","Type":"ContainerDied","Data":"ab7536cd8f3af8d7bfb7c13226d84cf3fd71979c255a4e23eae07e38f8de0cd6"}
Sep 17 00:40:12.025263 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.024632    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg" event={"ID":"e533b075-a947-45b0-bbbf-1ac95ade2554","Type":"ContainerDied","Data":"7a8edd98336e4539604ea477114eef81e4d9994ba08d385e37ef21615c09a138"}
Sep 17 00:40:12.025263 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.024678    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"
Sep 17 00:40:12.033202 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.033182    2577 scope.go:117] "RemoveContainer" containerID="aad81999c131fffef22c07c020895981989aa4c6fce7e8d5507eceef729a33ce"
Sep 17 00:40:12.033527 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:40:12.033502    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aad81999c131fffef22c07c020895981989aa4c6fce7e8d5507eceef729a33ce\": container with ID starting with aad81999c131fffef22c07c020895981989aa4c6fce7e8d5507eceef729a33ce not found: ID does not exist" containerID="aad81999c131fffef22c07c020895981989aa4c6fce7e8d5507eceef729a33ce"
Sep 17 00:40:12.033624 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.033536    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aad81999c131fffef22c07c020895981989aa4c6fce7e8d5507eceef729a33ce"} err="failed to get container status \"aad81999c131fffef22c07c020895981989aa4c6fce7e8d5507eceef729a33ce\": rpc error: code = NotFound desc = could not find container \"aad81999c131fffef22c07c020895981989aa4c6fce7e8d5507eceef729a33ce\": container with ID starting with aad81999c131fffef22c07c020895981989aa4c6fce7e8d5507eceef729a33ce not found: ID does not exist"
Sep 17 00:40:12.033624 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.033558    2577 scope.go:117] "RemoveContainer" containerID="3349f89d6ee5a70aba77cf304176b23e308f5462b2306735ec317d8ce18b10d6"
Sep 17 00:40:12.042640 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.042615    2577 scope.go:117] "RemoveContainer" containerID="3349f89d6ee5a70aba77cf304176b23e308f5462b2306735ec317d8ce18b10d6"
Sep 17 00:40:12.042971 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:40:12.042948    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3349f89d6ee5a70aba77cf304176b23e308f5462b2306735ec317d8ce18b10d6\": container with ID starting with 3349f89d6ee5a70aba77cf304176b23e308f5462b2306735ec317d8ce18b10d6 not found: ID does not exist" containerID="3349f89d6ee5a70aba77cf304176b23e308f5462b2306735ec317d8ce18b10d6"
Sep 17 00:40:12.043057 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.042980    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3349f89d6ee5a70aba77cf304176b23e308f5462b2306735ec317d8ce18b10d6"} err="failed to get container status \"3349f89d6ee5a70aba77cf304176b23e308f5462b2306735ec317d8ce18b10d6\": rpc error: code = NotFound desc = could not find container \"3349f89d6ee5a70aba77cf304176b23e308f5462b2306735ec317d8ce18b10d6\": container with ID starting with 3349f89d6ee5a70aba77cf304176b23e308f5462b2306735ec317d8ce18b10d6 not found: ID does not exist"
Sep 17 00:40:12.043057 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.043003    2577 scope.go:117] "RemoveContainer" containerID="ab7536cd8f3af8d7bfb7c13226d84cf3fd71979c255a4e23eae07e38f8de0cd6"
Sep 17 00:40:12.051489 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.051442    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"]
Sep 17 00:40:12.052284 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.052269    2577 scope.go:117] "RemoveContainer" containerID="ab7536cd8f3af8d7bfb7c13226d84cf3fd71979c255a4e23eae07e38f8de0cd6"
Sep 17 00:40:12.052563 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:40:12.052545    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab7536cd8f3af8d7bfb7c13226d84cf3fd71979c255a4e23eae07e38f8de0cd6\": container with ID starting with ab7536cd8f3af8d7bfb7c13226d84cf3fd71979c255a4e23eae07e38f8de0cd6 not found: ID does not exist" containerID="ab7536cd8f3af8d7bfb7c13226d84cf3fd71979c255a4e23eae07e38f8de0cd6"
Sep 17 00:40:12.052637 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.052567    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7536cd8f3af8d7bfb7c13226d84cf3fd71979c255a4e23eae07e38f8de0cd6"} err="failed to get container status \"ab7536cd8f3af8d7bfb7c13226d84cf3fd71979c255a4e23eae07e38f8de0cd6\": rpc error: code = NotFound desc = could not find container \"ab7536cd8f3af8d7bfb7c13226d84cf3fd71979c255a4e23eae07e38f8de0cd6\": container with ID starting with ab7536cd8f3af8d7bfb7c13226d84cf3fd71979c255a4e23eae07e38f8de0cd6 not found: ID does not exist"
Sep 17 00:40:12.055152 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.055130    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2faea3/distinct-2-w4-2faea3-kserve-b7ddbd5fd-4wvg6"]
Sep 17 00:40:12.064815 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.064795    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"]
Sep 17 00:40:12.067906 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.067887    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2faea3/simulator-w4-2faea3-kserve-8f8d854f5-6skdg"]
Sep 17 00:40:12.077381 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.077359    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"]
Sep 17 00:40:12.081337 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:12.081317    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2faea3/distinct-w4-2faea3-kserve-64678c55cc-dlgn8"]
Sep 17 00:40:13.607485 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:13.607455    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1886bc33-36ad-492e-a0c5-a2a317988c34" path="/var/lib/kubelet/pods/1886bc33-36ad-492e-a0c5-a2a317988c34/volumes"
Sep 17 00:40:13.607857 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:13.607845    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63be31f1-7be5-4cc6-a614-271adaa92912" path="/var/lib/kubelet/pods/63be31f1-7be5-4cc6-a614-271adaa92912/volumes"
Sep 17 00:40:13.608188 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:13.608175    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e533b075-a947-45b0-bbbf-1ac95ade2554" path="/var/lib/kubelet/pods/e533b075-a947-45b0-bbbf-1ac95ade2554/volumes"
Sep 17 00:40:22.599101 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:22.599062    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:40:36.797390 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:36.797352    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:40:46.604218 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:46.604174    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:40:48.414213 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414179    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7831-ffd729"]
Sep 17 00:40:48.414614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414514    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1886bc33-36ad-492e-a0c5-a2a317988c34" containerName="main"
Sep 17 00:40:48.414614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414526    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1886bc33-36ad-492e-a0c5-a2a317988c34" containerName="main"
Sep 17 00:40:48.414614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414534    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6ac5c53-01a8-431c-bd96-e8e6ebe5490b" containerName="maas-api"
Sep 17 00:40:48.414614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414540    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6ac5c53-01a8-431c-bd96-e8e6ebe5490b" containerName="maas-api"
Sep 17 00:40:48.414614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414550    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95df6c2c-6af3-44c2-bed7-399abf2d29e9" containerName="maas-api"
Sep 17 00:40:48.414614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414555    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="95df6c2c-6af3-44c2-bed7-399abf2d29e9" containerName="maas-api"
Sep 17 00:40:48.414614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414561    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b86d40c8-5566-45fe-a649-faf8a7231e95" containerName="maas-api"
Sep 17 00:40:48.414614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414568    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b86d40c8-5566-45fe-a649-faf8a7231e95" containerName="maas-api"
Sep 17 00:40:48.414614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414582    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2455bef-2bec-4e2e-9aed-f3734c47b8c9" containerName="main"
Sep 17 00:40:48.414614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414587    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2455bef-2bec-4e2e-9aed-f3734c47b8c9" containerName="main"
Sep 17 00:40:48.414614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414597    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e533b075-a947-45b0-bbbf-1ac95ade2554" containerName="main"
Sep 17 00:40:48.414614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414602    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e533b075-a947-45b0-bbbf-1ac95ade2554" containerName="main"
Sep 17 00:40:48.414614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414611    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f93433d-4c9e-40e1-8108-410cee61d3eb" containerName="main"
Sep 17 00:40:48.414614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414616    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f93433d-4c9e-40e1-8108-410cee61d3eb" containerName="main"
Sep 17 00:40:48.414614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414622    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97a1c787-acbc-4d1c-9657-b79b9db6de85" containerName="main"
Sep 17 00:40:48.415082 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414627    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="97a1c787-acbc-4d1c-9657-b79b9db6de85" containerName="main"
Sep 17 00:40:48.415082 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414633    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63be31f1-7be5-4cc6-a614-271adaa92912" containerName="main"
Sep 17 00:40:48.415082 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414638    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="63be31f1-7be5-4cc6-a614-271adaa92912" containerName="main"
Sep 17 00:40:48.415082 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414689    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="95df6c2c-6af3-44c2-bed7-399abf2d29e9" containerName="maas-api"
Sep 17 00:40:48.415082 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414697    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2455bef-2bec-4e2e-9aed-f3734c47b8c9" containerName="main"
Sep 17 00:40:48.415082 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414703    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6ac5c53-01a8-431c-bd96-e8e6ebe5490b" containerName="maas-api"
Sep 17 00:40:48.415082 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414709    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="97a1c787-acbc-4d1c-9657-b79b9db6de85" containerName="main"
Sep 17 00:40:48.415082 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414714    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f93433d-4c9e-40e1-8108-410cee61d3eb" containerName="main"
Sep 17 00:40:48.415082 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414720    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b86d40c8-5566-45fe-a649-faf8a7231e95" containerName="maas-api"
Sep 17 00:40:48.415082 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414725    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1886bc33-36ad-492e-a0c5-a2a317988c34" containerName="main"
Sep 17 00:40:48.415082 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414730    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="63be31f1-7be5-4cc6-a614-271adaa92912" containerName="main"
Sep 17 00:40:48.415082 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.414735    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e533b075-a947-45b0-bbbf-1ac95ade2554" containerName="main"
Sep 17 00:40:48.419167 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.419146    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7831-ffd729"
Sep 17 00:40:48.421734 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.421682    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hf767\""
Sep 17 00:40:48.422468 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.422441    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7831-ffd729"]
Sep 17 00:40:48.540921 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.540878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw2cw\" (UniqueName: \"kubernetes.io/projected/eade1f13-f1f9-4c70-858e-24c49332e37a-kube-api-access-bw2cw\") pod \"test-curl-7831-ffd729\" (UID: \"eade1f13-f1f9-4c70-858e-24c49332e37a\") " pod="opendatahub/test-curl-7831-ffd729"
Sep 17 00:40:48.642437 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.642390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw2cw\" (UniqueName: \"kubernetes.io/projected/eade1f13-f1f9-4c70-858e-24c49332e37a-kube-api-access-bw2cw\") pod \"test-curl-7831-ffd729\" (UID: \"eade1f13-f1f9-4c70-858e-24c49332e37a\") " pod="opendatahub/test-curl-7831-ffd729"
Sep 17 00:40:48.650124 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.650090    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw2cw\" (UniqueName: \"kubernetes.io/projected/eade1f13-f1f9-4c70-858e-24c49332e37a-kube-api-access-bw2cw\") pod \"test-curl-7831-ffd729\" (UID: \"eade1f13-f1f9-4c70-858e-24c49332e37a\") " pod="opendatahub/test-curl-7831-ffd729"
Sep 17 00:40:48.729434 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.729384    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7831-ffd729"
Sep 17 00:40:48.999853 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:48.999827    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7831-ffd729"]
Sep 17 00:40:49.001659 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:40:49.001635    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeade1f13_f1f9_4c70_858e_24c49332e37a.slice/crio-8030339be6c3b989f11c41266f89f418b98119854983069be0b6d63746cf5d45 WatchSource:0}: Error finding container 8030339be6c3b989f11c41266f89f418b98119854983069be0b6d63746cf5d45: Status 404 returned error can't find the container with id 8030339be6c3b989f11c41266f89f418b98119854983069be0b6d63746cf5d45
Sep 17 00:40:49.228630 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:49.228589    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7831-ffd729" event={"ID":"eade1f13-f1f9-4c70-858e-24c49332e37a","Type":"ContainerStarted","Data":"4a40f797b058768a22c6080a249c78729944702a702e15ec39b86ce64ba82b59"}
Sep 17 00:40:49.228799 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:49.228638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7831-ffd729" event={"ID":"eade1f13-f1f9-4c70-858e-24c49332e37a","Type":"ContainerStarted","Data":"8030339be6c3b989f11c41266f89f418b98119854983069be0b6d63746cf5d45"}
Sep 17 00:40:49.242171 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:49.242122    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7831-ffd729" podStartSLOduration=1.2421057 podStartE2EDuration="1.2421057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:40:49.241768851 +0000 UTC m=+1862.100261988" watchObservedRunningTime="2026-09-17 00:40:49.2421057 +0000 UTC m=+1862.100598835"
Sep 17 00:40:49.819071 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:49.819039    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7831-ffd729"]
Sep 17 00:40:49.821623 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:49.821597    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7831-ffd729"]
Sep 17 00:40:50.234469 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:40:50.234439    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7831-ffd729" container="test-curl-7831-ffd729"
Sep 17 00:40:51.245756 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:51.245712    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7831-ffd729" podUID="eade1f13-f1f9-4c70-858e-24c49332e37a" containerName="test-curl-7831-ffd729" containerID="cri-o://4a40f797b058768a22c6080a249c78729944702a702e15ec39b86ce64ba82b59" gracePeriod=2
Sep 17 00:40:51.247961 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:40:51.247926    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7831-ffd729\" is forbidden: User \"system:node:ip-10-0-139-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-27.ec2.internal' and this object" podUID="eade1f13-f1f9-4c70-858e-24c49332e37a" pod="opendatahub/test-curl-7831-ffd729"
Sep 17 00:40:52.180242 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:52.180199    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"]
Sep 17 00:40:53.385896 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:53.385869    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7831-ffd729"
Sep 17 00:40:53.486229 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:53.486194    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw2cw\" (UniqueName: \"kubernetes.io/projected/eade1f13-f1f9-4c70-858e-24c49332e37a-kube-api-access-bw2cw\") pod \"eade1f13-f1f9-4c70-858e-24c49332e37a\" (UID: \"eade1f13-f1f9-4c70-858e-24c49332e37a\") "
Sep 17 00:40:53.488286 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:53.488260    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eade1f13-f1f9-4c70-858e-24c49332e37a-kube-api-access-bw2cw" (OuterVolumeSpecName: "kube-api-access-bw2cw") pod "eade1f13-f1f9-4c70-858e-24c49332e37a" (UID: "eade1f13-f1f9-4c70-858e-24c49332e37a"). InnerVolumeSpecName "kube-api-access-bw2cw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:40:53.587369 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:53.587282    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw2cw\" (UniqueName: \"kubernetes.io/projected/eade1f13-f1f9-4c70-858e-24c49332e37a-kube-api-access-bw2cw\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:40:53.606941 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:53.606909    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eade1f13-f1f9-4c70-858e-24c49332e37a" path="/var/lib/kubelet/pods/eade1f13-f1f9-4c70-858e-24c49332e37a/volumes"
Sep 17 00:40:54.264994 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:54.264964    2577 generic.go:356] "Generic (PLEG): container finished" podID="eade1f13-f1f9-4c70-858e-24c49332e37a" containerID="4a40f797b058768a22c6080a249c78729944702a702e15ec39b86ce64ba82b59" exitCode=137
Sep 17 00:40:54.265193 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:54.265008    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7831-ffd729"
Sep 17 00:40:54.265193 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:54.265053    2577 scope.go:117] "RemoveContainer" containerID="4a40f797b058768a22c6080a249c78729944702a702e15ec39b86ce64ba82b59"
Sep 17 00:40:54.273601 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:54.273583    2577 scope.go:117] "RemoveContainer" containerID="4a40f797b058768a22c6080a249c78729944702a702e15ec39b86ce64ba82b59"
Sep 17 00:40:54.273880 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:40:54.273861    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a40f797b058768a22c6080a249c78729944702a702e15ec39b86ce64ba82b59\": container with ID starting with 4a40f797b058768a22c6080a249c78729944702a702e15ec39b86ce64ba82b59 not found: ID does not exist" containerID="4a40f797b058768a22c6080a249c78729944702a702e15ec39b86ce64ba82b59"
Sep 17 00:40:54.273923 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:54.273887    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a40f797b058768a22c6080a249c78729944702a702e15ec39b86ce64ba82b59"} err="failed to get container status \"4a40f797b058768a22c6080a249c78729944702a702e15ec39b86ce64ba82b59\": rpc error: code = NotFound desc = could not find container \"4a40f797b058768a22c6080a249c78729944702a702e15ec39b86ce64ba82b59\": container with ID starting with 4a40f797b058768a22c6080a249c78729944702a702e15ec39b86ce64ba82b59 not found: ID does not exist"
Sep 17 00:40:58.595116 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:40:58.595072    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:41:07.199162 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:07.199094    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp" podUID="602e7a60-9b84-4f02-8595-db8827604955" containerName="main" containerID="cri-o://202f74b7d899d249e94b455c675d88caa4495d8dcb9942235380769ce18e8d42" gracePeriod=45
Sep 17 00:41:07.332863 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:07.332832    2577 generic.go:356] "Generic (PLEG): container finished" podID="602e7a60-9b84-4f02-8595-db8827604955" containerID="202f74b7d899d249e94b455c675d88caa4495d8dcb9942235380769ce18e8d42" exitCode=0
Sep 17 00:41:07.333025 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:07.332909    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp" event={"ID":"602e7a60-9b84-4f02-8595-db8827604955","Type":"ContainerDied","Data":"202f74b7d899d249e94b455c675d88caa4495d8dcb9942235380769ce18e8d42"}
Sep 17 00:41:07.441937 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:07.441912    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:41:07.513647 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:07.513562    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/602e7a60-9b84-4f02-8595-db8827604955-dshm\") pod \"602e7a60-9b84-4f02-8595-db8827604955\" (UID: \"602e7a60-9b84-4f02-8595-db8827604955\") "
Sep 17 00:41:07.513647 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:07.513606    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/602e7a60-9b84-4f02-8595-db8827604955-home\") pod \"602e7a60-9b84-4f02-8595-db8827604955\" (UID: \"602e7a60-9b84-4f02-8595-db8827604955\") "
Sep 17 00:41:07.513888 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:07.513655    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/602e7a60-9b84-4f02-8595-db8827604955-tls-certs\") pod \"602e7a60-9b84-4f02-8595-db8827604955\" (UID: \"602e7a60-9b84-4f02-8595-db8827604955\") "
Sep 17 00:41:07.513888 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:07.513708    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/602e7a60-9b84-4f02-8595-db8827604955-tmp-dir\") pod \"602e7a60-9b84-4f02-8595-db8827604955\" (UID: \"602e7a60-9b84-4f02-8595-db8827604955\") "
Sep 17 00:41:07.513888 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:07.513773    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/602e7a60-9b84-4f02-8595-db8827604955-model-cache\") pod \"602e7a60-9b84-4f02-8595-db8827604955\" (UID: \"602e7a60-9b84-4f02-8595-db8827604955\") "
Sep 17 00:41:07.513888 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:07.513808    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szpjf\" (UniqueName: \"kubernetes.io/projected/602e7a60-9b84-4f02-8595-db8827604955-kube-api-access-szpjf\") pod \"602e7a60-9b84-4f02-8595-db8827604955\" (UID: \"602e7a60-9b84-4f02-8595-db8827604955\") "
Sep 17 00:41:07.514108 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:07.513948    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602e7a60-9b84-4f02-8595-db8827604955-home" (OuterVolumeSpecName: "home") pod "602e7a60-9b84-4f02-8595-db8827604955" (UID: "602e7a60-9b84-4f02-8595-db8827604955"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:41:07.514108 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:07.514002    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602e7a60-9b84-4f02-8595-db8827604955-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "602e7a60-9b84-4f02-8595-db8827604955" (UID: "602e7a60-9b84-4f02-8595-db8827604955"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:41:07.514108 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:07.514028    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602e7a60-9b84-4f02-8595-db8827604955-model-cache" (OuterVolumeSpecName: "model-cache") pod "602e7a60-9b84-4f02-8595-db8827604955" (UID: "602e7a60-9b84-4f02-8595-db8827604955"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:41:07.514108 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:07.514107    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/602e7a60-9b84-4f02-8595-db8827604955-tmp-dir\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:41:07.514331 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:07.514120    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/602e7a60-9b84-4f02-8595-db8827604955-model-cache\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:41:07.514331 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:07.514133    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/602e7a60-9b84-4f02-8595-db8827604955-home\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:41:07.515926 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:07.515887    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/602e7a60-9b84-4f02-8595-db8827604955-kube-api-access-szpjf" (OuterVolumeSpecName: "kube-api-access-szpjf") pod "602e7a60-9b84-4f02-8595-db8827604955" (UID: "602e7a60-9b84-4f02-8595-db8827604955"). InnerVolumeSpecName "kube-api-access-szpjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:41:07.515926 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:07.515919    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/602e7a60-9b84-4f02-8595-db8827604955-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "602e7a60-9b84-4f02-8595-db8827604955" (UID: "602e7a60-9b84-4f02-8595-db8827604955"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:41:07.516074 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:07.515925    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602e7a60-9b84-4f02-8595-db8827604955-dshm" (OuterVolumeSpecName: "dshm") pod "602e7a60-9b84-4f02-8595-db8827604955" (UID: "602e7a60-9b84-4f02-8595-db8827604955"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:41:07.614784 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:07.614760    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szpjf\" (UniqueName: \"kubernetes.io/projected/602e7a60-9b84-4f02-8595-db8827604955-kube-api-access-szpjf\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:41:07.614784 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:07.614785    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/602e7a60-9b84-4f02-8595-db8827604955-dshm\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:41:07.614932 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:07.614794    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/602e7a60-9b84-4f02-8595-db8827604955-tls-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:41:08.339050 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:08.339017    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"
Sep 17 00:41:08.339531 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:08.339017    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp" event={"ID":"602e7a60-9b84-4f02-8595-db8827604955","Type":"ContainerDied","Data":"744ea17c1aea68b790db6ba07812e805fe837fe2d4662710c84b64d1b0013fd8"}
Sep 17 00:41:08.339531 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:08.339148    2577 scope.go:117] "RemoveContainer" containerID="202f74b7d899d249e94b455c675d88caa4495d8dcb9942235380769ce18e8d42"
Sep 17 00:41:08.355283 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:08.355253    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"]
Sep 17 00:41:08.359501 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:08.359477    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-07f9f2/auth-test-model-07f9f2-kserve-5c5f7bcb4c-4m7bp"]
Sep 17 00:41:09.607455 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:09.607391    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="602e7a60-9b84-4f02-8595-db8827604955" path="/var/lib/kubelet/pods/602e7a60-9b84-4f02-8595-db8827604955/volumes"
Sep 17 00:41:26.297106 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:26.297069    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:41:43.404021 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.403985    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"]
Sep 17 00:41:43.404601 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.404562    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eade1f13-f1f9-4c70-858e-24c49332e37a" containerName="test-curl-7831-ffd729"
Sep 17 00:41:43.404601 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.404587    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eade1f13-f1f9-4c70-858e-24c49332e37a" containerName="test-curl-7831-ffd729"
Sep 17 00:41:43.404713 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.404606    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="602e7a60-9b84-4f02-8595-db8827604955" containerName="main"
Sep 17 00:41:43.404713 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.404612    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="602e7a60-9b84-4f02-8595-db8827604955" containerName="main"
Sep 17 00:41:43.404713 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.404693    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="602e7a60-9b84-4f02-8595-db8827604955" containerName="main"
Sep 17 00:41:43.404713 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.404702    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eade1f13-f1f9-4c70-858e-24c49332e37a" containerName="test-curl-7831-ffd729"
Sep 17 00:41:43.407855 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.407836    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:41:43.411246 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.411214    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-07f9f2\"/\"sub-test-model-07f9f2-kserve-self-signed-certs\""
Sep 17 00:41:43.411246 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.411233    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-07f9f2\"/\"kube-root-ca.crt\""
Sep 17 00:41:43.411470 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.411325    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-07f9f2\"/\"openshift-service-ca.crt\""
Sep 17 00:41:43.411470 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.411375    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-07f9f2\"/\"default-dockercfg-tp66z\""
Sep 17 00:41:43.416016 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.415995    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"]
Sep 17 00:41:43.426243 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.426218    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-tls-certs\") pod \"sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5\" (UID: \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:41:43.426377 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.426248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-tmp-dir\") pod \"sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5\" (UID: \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:41:43.426377 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.426274    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-model-cache\") pod \"sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5\" (UID: \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:41:43.426377 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.426316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-home\") pod \"sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5\" (UID: \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:41:43.426377 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.426345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcrkn\" (UniqueName: \"kubernetes.io/projected/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-kube-api-access-kcrkn\") pod \"sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5\" (UID: \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:41:43.426671 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.426389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-dshm\") pod \"sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5\" (UID: \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:41:43.529966 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.529919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-tls-certs\") pod \"sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5\" (UID: \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:41:43.530154 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.529978    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-tmp-dir\") pod \"sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5\" (UID: \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:41:43.530154 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.530006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-model-cache\") pod \"sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5\" (UID: \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:41:43.530154 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.530040    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-home\") pod \"sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5\" (UID: \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:41:43.530154 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.530083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcrkn\" (UniqueName: \"kubernetes.io/projected/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-kube-api-access-kcrkn\") pod \"sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5\" (UID: \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:41:43.530154 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.530125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-dshm\") pod \"sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5\" (UID: \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:41:43.530538 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.530506    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-tmp-dir\") pod \"sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5\" (UID: \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:41:43.530688 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.530651    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-home\") pod \"sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5\" (UID: \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:41:43.530841 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.530821    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-model-cache\") pod \"sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5\" (UID: \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:41:43.532742 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.532718    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-tls-certs\") pod \"sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5\" (UID: \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:41:43.532861 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.532840    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-dshm\") pod \"sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5\" (UID: \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:41:43.538923 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.538900    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcrkn\" (UniqueName: \"kubernetes.io/projected/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-kube-api-access-kcrkn\") pod \"sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5\" (UID: \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\") " pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:41:43.717751 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.717719    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:41:43.993097 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:43.993072    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"]
Sep 17 00:41:43.994195 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:41:43.994171    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1d4e264_8043_4d9a_9b87_47a152f5b2d9.slice/crio-6a5d5c515249f3823c927b710ed31ac04664bdb6cb49d665d36b0331e4ee1171 WatchSource:0}: Error finding container 6a5d5c515249f3823c927b710ed31ac04664bdb6cb49d665d36b0331e4ee1171: Status 404 returned error can't find the container with id 6a5d5c515249f3823c927b710ed31ac04664bdb6cb49d665d36b0331e4ee1171
Sep 17 00:41:44.539265 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:44.539210    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5" event={"ID":"c1d4e264-8043-4d9a-9b87-47a152f5b2d9","Type":"ContainerStarted","Data":"359195464ca989f54949d2996104674374d9a2ae54d45b523bfd2782681d9e12"}
Sep 17 00:41:44.539689 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:44.539280    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5" event={"ID":"c1d4e264-8043-4d9a-9b87-47a152f5b2d9","Type":"ContainerStarted","Data":"6a5d5c515249f3823c927b710ed31ac04664bdb6cb49d665d36b0331e4ee1171"}
Sep 17 00:41:44.557767 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:44.557715    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5" podStartSLOduration=1.55770063 podStartE2EDuration="1.55770063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:41:44.555402103 +0000 UTC m=+1917.413895273" watchObservedRunningTime="2026-09-17 00:41:44.55770063 +0000 UTC m=+1917.416193765"
Sep 17 00:41:47.001894 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:47.001844    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:41:53.718673 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:53.718632    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:41:53.719080 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:53.718686    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:41:53.731385 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:53.731355    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:41:54.604115 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:41:54.604080    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:42:00.300611 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:00.300574    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:42:08.732733 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:08.732688    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7831-063481"]
Sep 17 00:42:08.736094 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:08.736074    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7831-063481"
Sep 17 00:42:08.738586 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:08.738561    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hf767\""
Sep 17 00:42:08.741575 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:08.741556    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7831-063481"]
Sep 17 00:42:08.851559 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:08.851514    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvz8k\" (UniqueName: \"kubernetes.io/projected/7c837a57-2820-473a-980c-9f4ff1ac4426-kube-api-access-fvz8k\") pod \"test-curl-7831-063481\" (UID: \"7c837a57-2820-473a-980c-9f4ff1ac4426\") " pod="opendatahub/test-curl-7831-063481"
Sep 17 00:42:08.952880 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:08.952843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvz8k\" (UniqueName: \"kubernetes.io/projected/7c837a57-2820-473a-980c-9f4ff1ac4426-kube-api-access-fvz8k\") pod \"test-curl-7831-063481\" (UID: \"7c837a57-2820-473a-980c-9f4ff1ac4426\") " pod="opendatahub/test-curl-7831-063481"
Sep 17 00:42:08.960182 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:08.960146    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvz8k\" (UniqueName: \"kubernetes.io/projected/7c837a57-2820-473a-980c-9f4ff1ac4426-kube-api-access-fvz8k\") pod \"test-curl-7831-063481\" (UID: \"7c837a57-2820-473a-980c-9f4ff1ac4426\") " pod="opendatahub/test-curl-7831-063481"
Sep 17 00:42:09.047312 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:09.047231    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7831-063481"
Sep 17 00:42:09.315605 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:09.315579    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7831-063481"]
Sep 17 00:42:09.317570 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:42:09.317535    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c837a57_2820_473a_980c_9f4ff1ac4426.slice/crio-a11df2632189bcbca2d3c2ebd5067d8bb3da86f176b161cd17165eb04eccbf8c WatchSource:0}: Error finding container a11df2632189bcbca2d3c2ebd5067d8bb3da86f176b161cd17165eb04eccbf8c: Status 404 returned error can't find the container with id a11df2632189bcbca2d3c2ebd5067d8bb3da86f176b161cd17165eb04eccbf8c
Sep 17 00:42:09.675170 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:09.675074    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7831-063481" event={"ID":"7c837a57-2820-473a-980c-9f4ff1ac4426","Type":"ContainerStarted","Data":"0157ffd51f1bcabd16a1699340ad206a7eaac11d7bc734a4b5e135283f16477c"}
Sep 17 00:42:09.675170 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:09.675126    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7831-063481" event={"ID":"7c837a57-2820-473a-980c-9f4ff1ac4426","Type":"ContainerStarted","Data":"a11df2632189bcbca2d3c2ebd5067d8bb3da86f176b161cd17165eb04eccbf8c"}
Sep 17 00:42:09.688446 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:09.688357    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7831-063481" podStartSLOduration=1.6883367470000001 podStartE2EDuration="1.688336747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:42:09.68751141 +0000 UTC m=+1942.546004546" watchObservedRunningTime="2026-09-17 00:42:09.688336747 +0000 UTC m=+1942.546829883"
Sep 17 00:42:10.269149 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:10.269107    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7831-063481"]
Sep 17 00:42:10.272979 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:10.272948    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7831-063481"]
Sep 17 00:42:10.394772 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:10.394691    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7831-066533"]
Sep 17 00:42:10.395067 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:10.395053    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c837a57-2820-473a-980c-9f4ff1ac4426" containerName="test-curl-7831-063481"
Sep 17 00:42:10.395118 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:10.395068    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c837a57-2820-473a-980c-9f4ff1ac4426" containerName="test-curl-7831-063481"
Sep 17 00:42:10.395154 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:10.395122    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c837a57-2820-473a-980c-9f4ff1ac4426" containerName="test-curl-7831-063481"
Sep 17 00:42:10.398118 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:10.398101    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7831-066533"
Sep 17 00:42:10.401809 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:10.401788    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7831-066533"]
Sep 17 00:42:10.567846 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:10.567808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cz6d\" (UniqueName: \"kubernetes.io/projected/db7cc939-31c4-4d15-81bf-8b0118974236-kube-api-access-9cz6d\") pod \"test-curl-7831-066533\" (UID: \"db7cc939-31c4-4d15-81bf-8b0118974236\") " pod="opendatahub/test-curl-7831-066533"
Sep 17 00:42:10.668318 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:10.668236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cz6d\" (UniqueName: \"kubernetes.io/projected/db7cc939-31c4-4d15-81bf-8b0118974236-kube-api-access-9cz6d\") pod \"test-curl-7831-066533\" (UID: \"db7cc939-31c4-4d15-81bf-8b0118974236\") " pod="opendatahub/test-curl-7831-066533"
Sep 17 00:42:10.676028 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:10.675986    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cz6d\" (UniqueName: \"kubernetes.io/projected/db7cc939-31c4-4d15-81bf-8b0118974236-kube-api-access-9cz6d\") pod \"test-curl-7831-066533\" (UID: \"db7cc939-31c4-4d15-81bf-8b0118974236\") " pod="opendatahub/test-curl-7831-066533"
Sep 17 00:42:10.680720 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:42:10.680698    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7831-063481" container="test-curl-7831-063481"
Sep 17 00:42:10.708653 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:10.708633    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7831-066533"
Sep 17 00:42:10.977129 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:10.977099    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7831-066533"]
Sep 17 00:42:10.979229 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:42:10.979200    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb7cc939_31c4_4d15_81bf_8b0118974236.slice/crio-370c6fbfc1a5344f257c399687024c33161b76257415ce44caf3750ef4cd7da0 WatchSource:0}: Error finding container 370c6fbfc1a5344f257c399687024c33161b76257415ce44caf3750ef4cd7da0: Status 404 returned error can't find the container with id 370c6fbfc1a5344f257c399687024c33161b76257415ce44caf3750ef4cd7da0
Sep 17 00:42:11.687928 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:11.687891    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7831-066533" event={"ID":"db7cc939-31c4-4d15-81bf-8b0118974236","Type":"ContainerStarted","Data":"bbe20baacc058a248b1ce42cae00470d461215aafba760090dd6e53cfec9babb"}
Sep 17 00:42:11.688448 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:11.687914    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7831-063481" podUID="7c837a57-2820-473a-980c-9f4ff1ac4426" containerName="test-curl-7831-063481" containerID="cri-o://0157ffd51f1bcabd16a1699340ad206a7eaac11d7bc734a4b5e135283f16477c" gracePeriod=2
Sep 17 00:42:11.688448 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:11.687940    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7831-066533" event={"ID":"db7cc939-31c4-4d15-81bf-8b0118974236","Type":"ContainerStarted","Data":"370c6fbfc1a5344f257c399687024c33161b76257415ce44caf3750ef4cd7da0"}
Sep 17 00:42:11.701753 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:11.701706    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7831-066533" podStartSLOduration=1.7016920899999999 podStartE2EDuration="1.70169209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:42:11.700968917 +0000 UTC m=+1944.559462052" watchObservedRunningTime="2026-09-17 00:42:11.70169209 +0000 UTC m=+1944.560185225"
Sep 17 00:42:12.262045 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:12.262008    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7831-066533"]
Sep 17 00:42:12.264306 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:12.264277    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7831-066533"]
Sep 17 00:42:12.692992 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:42:12.692955    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7831-066533" container="test-curl-7831-066533"
Sep 17 00:42:13.699301 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:13.699266    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7831-066533" podUID="db7cc939-31c4-4d15-81bf-8b0118974236" containerName="test-curl-7831-066533" containerID="cri-o://bbe20baacc058a248b1ce42cae00470d461215aafba760090dd6e53cfec9babb" gracePeriod=2
Sep 17 00:42:13.872659 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:13.872627    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7831-063481"
Sep 17 00:42:13.996148 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:13.996051    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvz8k\" (UniqueName: \"kubernetes.io/projected/7c837a57-2820-473a-980c-9f4ff1ac4426-kube-api-access-fvz8k\") pod \"7c837a57-2820-473a-980c-9f4ff1ac4426\" (UID: \"7c837a57-2820-473a-980c-9f4ff1ac4426\") "
Sep 17 00:42:13.998139 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:13.998107    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c837a57-2820-473a-980c-9f4ff1ac4426-kube-api-access-fvz8k" (OuterVolumeSpecName: "kube-api-access-fvz8k") pod "7c837a57-2820-473a-980c-9f4ff1ac4426" (UID: "7c837a57-2820-473a-980c-9f4ff1ac4426"). InnerVolumeSpecName "kube-api-access-fvz8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:42:14.096929 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:14.096886    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvz8k\" (UniqueName: \"kubernetes.io/projected/7c837a57-2820-473a-980c-9f4ff1ac4426-kube-api-access-fvz8k\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:42:14.705235 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:14.705200    2577 generic.go:356] "Generic (PLEG): container finished" podID="7c837a57-2820-473a-980c-9f4ff1ac4426" containerID="0157ffd51f1bcabd16a1699340ad206a7eaac11d7bc734a4b5e135283f16477c" exitCode=137
Sep 17 00:42:14.705845 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:14.705246    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7831-063481"
Sep 17 00:42:14.705845 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:14.705262    2577 scope.go:117] "RemoveContainer" containerID="0157ffd51f1bcabd16a1699340ad206a7eaac11d7bc734a4b5e135283f16477c"
Sep 17 00:42:14.714169 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:14.714152    2577 scope.go:117] "RemoveContainer" containerID="0157ffd51f1bcabd16a1699340ad206a7eaac11d7bc734a4b5e135283f16477c"
Sep 17 00:42:14.714463 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:42:14.714443    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0157ffd51f1bcabd16a1699340ad206a7eaac11d7bc734a4b5e135283f16477c\": container with ID starting with 0157ffd51f1bcabd16a1699340ad206a7eaac11d7bc734a4b5e135283f16477c not found: ID does not exist" containerID="0157ffd51f1bcabd16a1699340ad206a7eaac11d7bc734a4b5e135283f16477c"
Sep 17 00:42:14.714509 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:14.714472    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0157ffd51f1bcabd16a1699340ad206a7eaac11d7bc734a4b5e135283f16477c"} err="failed to get container status \"0157ffd51f1bcabd16a1699340ad206a7eaac11d7bc734a4b5e135283f16477c\": rpc error: code = NotFound desc = could not find container \"0157ffd51f1bcabd16a1699340ad206a7eaac11d7bc734a4b5e135283f16477c\": container with ID starting with 0157ffd51f1bcabd16a1699340ad206a7eaac11d7bc734a4b5e135283f16477c not found: ID does not exist"
Sep 17 00:42:14.715759 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:42:14.715736    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7831-063481\" is forbidden: User \"system:node:ip-10-0-139-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-27.ec2.internal' and this object" podUID="7c837a57-2820-473a-980c-9f4ff1ac4426" pod="opendatahub/test-curl-7831-063481"
Sep 17 00:42:15.297038 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:15.296990    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"]
Sep 17 00:42:15.299437 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:42:15.299385    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7831-063481\" is forbidden: User \"system:node:ip-10-0-139-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-27.ec2.internal' and this object" podUID="7c837a57-2820-473a-980c-9f4ff1ac4426" pod="opendatahub/test-curl-7831-063481"
Sep 17 00:42:15.405319 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:15.405282    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:42:15.607459 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:15.607357    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c837a57-2820-473a-980c-9f4ff1ac4426" path="/var/lib/kubelet/pods/7c837a57-2820-473a-980c-9f4ff1ac4426/volumes"
Sep 17 00:42:15.840669 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:15.840644    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7831-066533"
Sep 17 00:42:16.010244 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:16.010207    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cz6d\" (UniqueName: \"kubernetes.io/projected/db7cc939-31c4-4d15-81bf-8b0118974236-kube-api-access-9cz6d\") pod \"db7cc939-31c4-4d15-81bf-8b0118974236\" (UID: \"db7cc939-31c4-4d15-81bf-8b0118974236\") "
Sep 17 00:42:16.012314 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:16.012279    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db7cc939-31c4-4d15-81bf-8b0118974236-kube-api-access-9cz6d" (OuterVolumeSpecName: "kube-api-access-9cz6d") pod "db7cc939-31c4-4d15-81bf-8b0118974236" (UID: "db7cc939-31c4-4d15-81bf-8b0118974236"). InnerVolumeSpecName "kube-api-access-9cz6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:42:16.111255 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:16.111212    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cz6d\" (UniqueName: \"kubernetes.io/projected/db7cc939-31c4-4d15-81bf-8b0118974236-kube-api-access-9cz6d\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:42:16.719332 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:16.719299    2577 generic.go:356] "Generic (PLEG): container finished" podID="db7cc939-31c4-4d15-81bf-8b0118974236" containerID="bbe20baacc058a248b1ce42cae00470d461215aafba760090dd6e53cfec9babb" exitCode=137
Sep 17 00:42:16.719562 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:16.719344    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7831-066533"
Sep 17 00:42:16.719562 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:16.719405    2577 scope.go:117] "RemoveContainer" containerID="bbe20baacc058a248b1ce42cae00470d461215aafba760090dd6e53cfec9babb"
Sep 17 00:42:16.728266 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:16.728245    2577 scope.go:117] "RemoveContainer" containerID="bbe20baacc058a248b1ce42cae00470d461215aafba760090dd6e53cfec9babb"
Sep 17 00:42:16.728580 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:42:16.728562    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbe20baacc058a248b1ce42cae00470d461215aafba760090dd6e53cfec9babb\": container with ID starting with bbe20baacc058a248b1ce42cae00470d461215aafba760090dd6e53cfec9babb not found: ID does not exist" containerID="bbe20baacc058a248b1ce42cae00470d461215aafba760090dd6e53cfec9babb"
Sep 17 00:42:16.728651 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:16.728594    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbe20baacc058a248b1ce42cae00470d461215aafba760090dd6e53cfec9babb"} err="failed to get container status \"bbe20baacc058a248b1ce42cae00470d461215aafba760090dd6e53cfec9babb\": rpc error: code = NotFound desc = could not find container \"bbe20baacc058a248b1ce42cae00470d461215aafba760090dd6e53cfec9babb\": container with ID starting with bbe20baacc058a248b1ce42cae00470d461215aafba760090dd6e53cfec9babb not found: ID does not exist"
Sep 17 00:42:16.729776 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:42:16.729752    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7831-066533\" is forbidden: User \"system:node:ip-10-0-139-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-27.ec2.internal' and this object" podUID="db7cc939-31c4-4d15-81bf-8b0118974236" pod="opendatahub/test-curl-7831-066533"
Sep 17 00:42:17.606128 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:42:17.606086    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7831-066533\" is forbidden: User \"system:node:ip-10-0-139-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-27.ec2.internal' and this object" podUID="db7cc939-31c4-4d15-81bf-8b0118974236" pod="opendatahub/test-curl-7831-066533"
Sep 17 00:42:17.607183 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:17.607163    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db7cc939-31c4-4d15-81bf-8b0118974236" path="/var/lib/kubelet/pods/db7cc939-31c4-4d15-81bf-8b0118974236/volumes"
Sep 17 00:42:22.399247 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.399205    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"]
Sep 17 00:42:22.399734 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.399715    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db7cc939-31c4-4d15-81bf-8b0118974236" containerName="test-curl-7831-066533"
Sep 17 00:42:22.399783 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.399736    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="db7cc939-31c4-4d15-81bf-8b0118974236" containerName="test-curl-7831-066533"
Sep 17 00:42:22.399820 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.399808    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="db7cc939-31c4-4d15-81bf-8b0118974236" containerName="test-curl-7831-066533"
Sep 17 00:42:22.402942 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.402916    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:42:22.406665 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.406641    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-47a2db\"/\"openshift-service-ca.crt\""
Sep 17 00:42:22.406793 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.406660    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-47a2db\"/\"kube-root-ca.crt\""
Sep 17 00:42:22.406793 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.406641    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-47a2db\"/\"default-dockercfg-6lfqt\""
Sep 17 00:42:22.406793 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.406641    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-47a2db\"/\"rate-test-model-47a2db-kserve-self-signed-certs\""
Sep 17 00:42:22.413280 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.413252    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"]
Sep 17 00:42:22.573801 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.573745    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3512c11c-023c-44fc-898e-1625fb34d521-dshm\") pod \"rate-test-model-47a2db-kserve-795d659587-z5j9w\" (UID: \"3512c11c-023c-44fc-898e-1625fb34d521\") " pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:42:22.573992 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.573820    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3512c11c-023c-44fc-898e-1625fb34d521-tls-certs\") pod \"rate-test-model-47a2db-kserve-795d659587-z5j9w\" (UID: \"3512c11c-023c-44fc-898e-1625fb34d521\") " pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:42:22.573992 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.573873    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzlhx\" (UniqueName: \"kubernetes.io/projected/3512c11c-023c-44fc-898e-1625fb34d521-kube-api-access-bzlhx\") pod \"rate-test-model-47a2db-kserve-795d659587-z5j9w\" (UID: \"3512c11c-023c-44fc-898e-1625fb34d521\") " pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:42:22.573992 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.573898    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3512c11c-023c-44fc-898e-1625fb34d521-model-cache\") pod \"rate-test-model-47a2db-kserve-795d659587-z5j9w\" (UID: \"3512c11c-023c-44fc-898e-1625fb34d521\") " pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:42:22.573992 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.573936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3512c11c-023c-44fc-898e-1625fb34d521-home\") pod \"rate-test-model-47a2db-kserve-795d659587-z5j9w\" (UID: \"3512c11c-023c-44fc-898e-1625fb34d521\") " pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:42:22.573992 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.573950    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3512c11c-023c-44fc-898e-1625fb34d521-tmp-dir\") pod \"rate-test-model-47a2db-kserve-795d659587-z5j9w\" (UID: \"3512c11c-023c-44fc-898e-1625fb34d521\") " pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:42:22.674750 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.674643    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3512c11c-023c-44fc-898e-1625fb34d521-home\") pod \"rate-test-model-47a2db-kserve-795d659587-z5j9w\" (UID: \"3512c11c-023c-44fc-898e-1625fb34d521\") " pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:42:22.674750 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.674698    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3512c11c-023c-44fc-898e-1625fb34d521-tmp-dir\") pod \"rate-test-model-47a2db-kserve-795d659587-z5j9w\" (UID: \"3512c11c-023c-44fc-898e-1625fb34d521\") " pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:42:22.675009 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.674756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3512c11c-023c-44fc-898e-1625fb34d521-dshm\") pod \"rate-test-model-47a2db-kserve-795d659587-z5j9w\" (UID: \"3512c11c-023c-44fc-898e-1625fb34d521\") " pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:42:22.675009 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.674802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3512c11c-023c-44fc-898e-1625fb34d521-tls-certs\") pod \"rate-test-model-47a2db-kserve-795d659587-z5j9w\" (UID: \"3512c11c-023c-44fc-898e-1625fb34d521\") " pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:42:22.675009 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.674840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzlhx\" (UniqueName: \"kubernetes.io/projected/3512c11c-023c-44fc-898e-1625fb34d521-kube-api-access-bzlhx\") pod \"rate-test-model-47a2db-kserve-795d659587-z5j9w\" (UID: \"3512c11c-023c-44fc-898e-1625fb34d521\") " pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:42:22.675009 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.674942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3512c11c-023c-44fc-898e-1625fb34d521-model-cache\") pod \"rate-test-model-47a2db-kserve-795d659587-z5j9w\" (UID: \"3512c11c-023c-44fc-898e-1625fb34d521\") " pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:42:22.675304 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.675037    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3512c11c-023c-44fc-898e-1625fb34d521-tmp-dir\") pod \"rate-test-model-47a2db-kserve-795d659587-z5j9w\" (UID: \"3512c11c-023c-44fc-898e-1625fb34d521\") " pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:42:22.675304 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.675074    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3512c11c-023c-44fc-898e-1625fb34d521-home\") pod \"rate-test-model-47a2db-kserve-795d659587-z5j9w\" (UID: \"3512c11c-023c-44fc-898e-1625fb34d521\") " pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:42:22.675304 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.675257    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3512c11c-023c-44fc-898e-1625fb34d521-model-cache\") pod \"rate-test-model-47a2db-kserve-795d659587-z5j9w\" (UID: \"3512c11c-023c-44fc-898e-1625fb34d521\") " pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:42:22.677185 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.677151    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3512c11c-023c-44fc-898e-1625fb34d521-dshm\") pod \"rate-test-model-47a2db-kserve-795d659587-z5j9w\" (UID: \"3512c11c-023c-44fc-898e-1625fb34d521\") " pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:42:22.677377 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.677354    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3512c11c-023c-44fc-898e-1625fb34d521-tls-certs\") pod \"rate-test-model-47a2db-kserve-795d659587-z5j9w\" (UID: \"3512c11c-023c-44fc-898e-1625fb34d521\") " pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:42:22.682350 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.682331    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzlhx\" (UniqueName: \"kubernetes.io/projected/3512c11c-023c-44fc-898e-1625fb34d521-kube-api-access-bzlhx\") pod \"rate-test-model-47a2db-kserve-795d659587-z5j9w\" (UID: \"3512c11c-023c-44fc-898e-1625fb34d521\") " pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:42:22.712512 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.712468    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:42:22.987722 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:22.987691    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"]
Sep 17 00:42:22.991001 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:42:22.990967    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3512c11c_023c_44fc_898e_1625fb34d521.slice/crio-9a1583d2a96c003a5bae01af0f33c1eca9b7e8cd1f69f96729280954bd06de4c WatchSource:0}: Error finding container 9a1583d2a96c003a5bae01af0f33c1eca9b7e8cd1f69f96729280954bd06de4c: Status 404 returned error can't find the container with id 9a1583d2a96c003a5bae01af0f33c1eca9b7e8cd1f69f96729280954bd06de4c
Sep 17 00:42:23.761292 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:23.761249    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w" event={"ID":"3512c11c-023c-44fc-898e-1625fb34d521","Type":"ContainerStarted","Data":"b423d4bcaffa6ba4958bfbcc652983db290532c84732b8ed6be635d949a4e354"}
Sep 17 00:42:23.761292 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:23.761297    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w" event={"ID":"3512c11c-023c-44fc-898e-1625fb34d521","Type":"ContainerStarted","Data":"9a1583d2a96c003a5bae01af0f33c1eca9b7e8cd1f69f96729280954bd06de4c"}
Sep 17 00:42:23.778226 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:23.778173    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w" podStartSLOduration=1.778151715 podStartE2EDuration="1.778151715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:42:23.776154977 +0000 UTC m=+1956.634648105" watchObservedRunningTime="2026-09-17 00:42:23.778151715 +0000 UTC m=+1956.636644849"
Sep 17 00:42:26.106605 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:26.106554    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:42:30.317469 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.317409    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5" podUID="c1d4e264-8043-4d9a-9b87-47a152f5b2d9" containerName="main" containerID="cri-o://359195464ca989f54949d2996104674374d9a2ae54d45b523bfd2782681d9e12" gracePeriod=45
Sep 17 00:42:30.564286 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.564258    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:42:30.638060 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.637963    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcrkn\" (UniqueName: \"kubernetes.io/projected/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-kube-api-access-kcrkn\") pod \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\" (UID: \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\") "
Sep 17 00:42:30.638060 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.638012    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-model-cache\") pod \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\" (UID: \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\") "
Sep 17 00:42:30.638060 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.638061    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-tls-certs\") pod \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\" (UID: \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\") "
Sep 17 00:42:30.638353 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.638212    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-tmp-dir\") pod \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\" (UID: \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\") "
Sep 17 00:42:30.638353 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.638245    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-home\") pod \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\" (UID: \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\") "
Sep 17 00:42:30.638353 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.638276    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "c1d4e264-8043-4d9a-9b87-47a152f5b2d9" (UID: "c1d4e264-8043-4d9a-9b87-47a152f5b2d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:42:30.638353 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.638291    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-dshm\") pod \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\" (UID: \"c1d4e264-8043-4d9a-9b87-47a152f5b2d9\") "
Sep 17 00:42:30.638554 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.638484    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-home" (OuterVolumeSpecName: "home") pod "c1d4e264-8043-4d9a-9b87-47a152f5b2d9" (UID: "c1d4e264-8043-4d9a-9b87-47a152f5b2d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:42:30.638554 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.638505    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c1d4e264-8043-4d9a-9b87-47a152f5b2d9" (UID: "c1d4e264-8043-4d9a-9b87-47a152f5b2d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:42:30.638638 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.638624    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-model-cache\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:42:30.638696 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.638648    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-tmp-dir\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:42:30.638696 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.638660    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-home\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:42:30.640169 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.640145    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c1d4e264-8043-4d9a-9b87-47a152f5b2d9" (UID: "c1d4e264-8043-4d9a-9b87-47a152f5b2d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:42:30.640247 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.640206    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-dshm" (OuterVolumeSpecName: "dshm") pod "c1d4e264-8043-4d9a-9b87-47a152f5b2d9" (UID: "c1d4e264-8043-4d9a-9b87-47a152f5b2d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:42:30.640247 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.640230    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-kube-api-access-kcrkn" (OuterVolumeSpecName: "kube-api-access-kcrkn") pod "c1d4e264-8043-4d9a-9b87-47a152f5b2d9" (UID: "c1d4e264-8043-4d9a-9b87-47a152f5b2d9"). InnerVolumeSpecName "kube-api-access-kcrkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:42:30.739610 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.739569    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-tls-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:42:30.739610 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.739603    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-dshm\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:42:30.739610 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.739617    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcrkn\" (UniqueName: \"kubernetes.io/projected/c1d4e264-8043-4d9a-9b87-47a152f5b2d9-kube-api-access-kcrkn\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:42:30.799992 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.799953    2577 generic.go:356] "Generic (PLEG): container finished" podID="c1d4e264-8043-4d9a-9b87-47a152f5b2d9" containerID="359195464ca989f54949d2996104674374d9a2ae54d45b523bfd2782681d9e12" exitCode=0
Sep 17 00:42:30.800187 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.800002    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"
Sep 17 00:42:30.800187 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.799999    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5" event={"ID":"c1d4e264-8043-4d9a-9b87-47a152f5b2d9","Type":"ContainerDied","Data":"359195464ca989f54949d2996104674374d9a2ae54d45b523bfd2782681d9e12"}
Sep 17 00:42:30.800187 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.800117    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5" event={"ID":"c1d4e264-8043-4d9a-9b87-47a152f5b2d9","Type":"ContainerDied","Data":"6a5d5c515249f3823c927b710ed31ac04664bdb6cb49d665d36b0331e4ee1171"}
Sep 17 00:42:30.800187 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.800145    2577 scope.go:117] "RemoveContainer" containerID="359195464ca989f54949d2996104674374d9a2ae54d45b523bfd2782681d9e12"
Sep 17 00:42:30.809483 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.809464    2577 scope.go:117] "RemoveContainer" containerID="359195464ca989f54949d2996104674374d9a2ae54d45b523bfd2782681d9e12"
Sep 17 00:42:30.809734 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:42:30.809713    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"359195464ca989f54949d2996104674374d9a2ae54d45b523bfd2782681d9e12\": container with ID starting with 359195464ca989f54949d2996104674374d9a2ae54d45b523bfd2782681d9e12 not found: ID does not exist" containerID="359195464ca989f54949d2996104674374d9a2ae54d45b523bfd2782681d9e12"
Sep 17 00:42:30.809777 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.809741    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359195464ca989f54949d2996104674374d9a2ae54d45b523bfd2782681d9e12"} err="failed to get container status \"359195464ca989f54949d2996104674374d9a2ae54d45b523bfd2782681d9e12\": rpc error: code = NotFound desc = could not find container \"359195464ca989f54949d2996104674374d9a2ae54d45b523bfd2782681d9e12\": container with ID starting with 359195464ca989f54949d2996104674374d9a2ae54d45b523bfd2782681d9e12 not found: ID does not exist"
Sep 17 00:42:30.820937 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.820913    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"]
Sep 17 00:42:30.823908 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:30.823888    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-07f9f2/sub-test-model-07f9f2-kserve-99f4bfff9-4ntb5"]
Sep 17 00:42:31.607704 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:31.607670    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1d4e264-8043-4d9a-9b87-47a152f5b2d9" path="/var/lib/kubelet/pods/c1d4e264-8043-4d9a-9b87-47a152f5b2d9/volumes"
Sep 17 00:42:32.712854 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:32.712810    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:42:32.713287 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:32.712868    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:42:32.725539 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:32.725509    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:42:32.824630 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:32.824597    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:42:47.600819 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:42:47.600732    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:43:07.902665 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:07.902628    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:43:17.202608 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:17.202568    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:43:21.991306 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:21.991267    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"]
Sep 17 00:43:28.296337 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:28.296299    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:43:37.012520 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:37.012455    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w" podUID="3512c11c-023c-44fc-898e-1625fb34d521" containerName="main" containerID="cri-o://b423d4bcaffa6ba4958bfbcc652983db290532c84732b8ed6be635d949a4e354" gracePeriod=45
Sep 17 00:43:37.150178 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:37.150138    2577 generic.go:356] "Generic (PLEG): container finished" podID="3512c11c-023c-44fc-898e-1625fb34d521" containerID="b423d4bcaffa6ba4958bfbcc652983db290532c84732b8ed6be635d949a4e354" exitCode=0
Sep 17 00:43:37.150345 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:37.150213    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w" event={"ID":"3512c11c-023c-44fc-898e-1625fb34d521","Type":"ContainerDied","Data":"b423d4bcaffa6ba4958bfbcc652983db290532c84732b8ed6be635d949a4e354"}
Sep 17 00:43:37.270082 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:37.270017    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:43:37.330264 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:37.330224    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3512c11c-023c-44fc-898e-1625fb34d521-dshm\") pod \"3512c11c-023c-44fc-898e-1625fb34d521\" (UID: \"3512c11c-023c-44fc-898e-1625fb34d521\") "
Sep 17 00:43:37.330493 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:37.330328    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3512c11c-023c-44fc-898e-1625fb34d521-tls-certs\") pod \"3512c11c-023c-44fc-898e-1625fb34d521\" (UID: \"3512c11c-023c-44fc-898e-1625fb34d521\") "
Sep 17 00:43:37.330493 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:37.330353    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3512c11c-023c-44fc-898e-1625fb34d521-tmp-dir\") pod \"3512c11c-023c-44fc-898e-1625fb34d521\" (UID: \"3512c11c-023c-44fc-898e-1625fb34d521\") "
Sep 17 00:43:37.330493 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:37.330396    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3512c11c-023c-44fc-898e-1625fb34d521-home\") pod \"3512c11c-023c-44fc-898e-1625fb34d521\" (UID: \"3512c11c-023c-44fc-898e-1625fb34d521\") "
Sep 17 00:43:37.330493 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:37.330476    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzlhx\" (UniqueName: \"kubernetes.io/projected/3512c11c-023c-44fc-898e-1625fb34d521-kube-api-access-bzlhx\") pod \"3512c11c-023c-44fc-898e-1625fb34d521\" (UID: \"3512c11c-023c-44fc-898e-1625fb34d521\") "
Sep 17 00:43:37.330718 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:37.330500    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3512c11c-023c-44fc-898e-1625fb34d521-model-cache\") pod \"3512c11c-023c-44fc-898e-1625fb34d521\" (UID: \"3512c11c-023c-44fc-898e-1625fb34d521\") "
Sep 17 00:43:37.330718 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:37.330648    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3512c11c-023c-44fc-898e-1625fb34d521-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3512c11c-023c-44fc-898e-1625fb34d521" (UID: "3512c11c-023c-44fc-898e-1625fb34d521"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:43:37.330834 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:37.330719    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3512c11c-023c-44fc-898e-1625fb34d521-home" (OuterVolumeSpecName: "home") pod "3512c11c-023c-44fc-898e-1625fb34d521" (UID: "3512c11c-023c-44fc-898e-1625fb34d521"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:43:37.330834 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:37.330782    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3512c11c-023c-44fc-898e-1625fb34d521-model-cache" (OuterVolumeSpecName: "model-cache") pod "3512c11c-023c-44fc-898e-1625fb34d521" (UID: "3512c11c-023c-44fc-898e-1625fb34d521"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:43:37.330834 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:37.330813    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3512c11c-023c-44fc-898e-1625fb34d521-tmp-dir\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:43:37.330834 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:37.330825    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3512c11c-023c-44fc-898e-1625fb34d521-home\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:43:37.332622 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:37.332598    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3512c11c-023c-44fc-898e-1625fb34d521-dshm" (OuterVolumeSpecName: "dshm") pod "3512c11c-023c-44fc-898e-1625fb34d521" (UID: "3512c11c-023c-44fc-898e-1625fb34d521"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:43:37.332722 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:37.332706    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3512c11c-023c-44fc-898e-1625fb34d521-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3512c11c-023c-44fc-898e-1625fb34d521" (UID: "3512c11c-023c-44fc-898e-1625fb34d521"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:43:37.332772 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:37.332724    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3512c11c-023c-44fc-898e-1625fb34d521-kube-api-access-bzlhx" (OuterVolumeSpecName: "kube-api-access-bzlhx") pod "3512c11c-023c-44fc-898e-1625fb34d521" (UID: "3512c11c-023c-44fc-898e-1625fb34d521"). InnerVolumeSpecName "kube-api-access-bzlhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:43:37.431388 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:37.431344    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzlhx\" (UniqueName: \"kubernetes.io/projected/3512c11c-023c-44fc-898e-1625fb34d521-kube-api-access-bzlhx\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:43:37.431388 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:37.431377    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3512c11c-023c-44fc-898e-1625fb34d521-model-cache\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:43:37.431388 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:37.431386    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3512c11c-023c-44fc-898e-1625fb34d521-dshm\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:43:37.431388 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:37.431394    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3512c11c-023c-44fc-898e-1625fb34d521-tls-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:43:38.156745 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:38.156716    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"
Sep 17 00:43:38.156745 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:38.156727    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w" event={"ID":"3512c11c-023c-44fc-898e-1625fb34d521","Type":"ContainerDied","Data":"9a1583d2a96c003a5bae01af0f33c1eca9b7e8cd1f69f96729280954bd06de4c"}
Sep 17 00:43:38.157292 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:38.156788    2577 scope.go:117] "RemoveContainer" containerID="b423d4bcaffa6ba4958bfbcc652983db290532c84732b8ed6be635d949a4e354"
Sep 17 00:43:38.171597 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:38.171570    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"]
Sep 17 00:43:38.176925 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:38.176901    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-47a2db/rate-test-model-47a2db-kserve-795d659587-z5j9w"]
Sep 17 00:43:38.897117 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:38.897075    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:43:39.607154 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:39.607121    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3512c11c-023c-44fc-898e-1625fb34d521" path="/var/lib/kubelet/pods/3512c11c-023c-44fc-898e-1625fb34d521/volumes"
Sep 17 00:43:58.710164 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:58.710120    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff"]
Sep 17 00:43:58.710717 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:58.710685    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3512c11c-023c-44fc-898e-1625fb34d521" containerName="main"
Sep 17 00:43:58.710717 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:58.710708    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3512c11c-023c-44fc-898e-1625fb34d521" containerName="main"
Sep 17 00:43:58.710844 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:58.710734    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1d4e264-8043-4d9a-9b87-47a152f5b2d9" containerName="main"
Sep 17 00:43:58.710844 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:58.710743    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1d4e264-8043-4d9a-9b87-47a152f5b2d9" containerName="main"
Sep 17 00:43:58.710844 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:58.710837    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1d4e264-8043-4d9a-9b87-47a152f5b2d9" containerName="main"
Sep 17 00:43:58.711009 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:58.710855    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3512c11c-023c-44fc-898e-1625fb34d521" containerName="main"
Sep 17 00:43:58.714297 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:58.714274    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff"
Sep 17 00:43:58.716764 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:58.716736    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a8402d-serving-cert\""
Sep 17 00:43:58.724884 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:58.724846    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff"]
Sep 17 00:43:58.820331 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:58.820291    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c107238c-2f65-454b-b5ce-57360fd751a0-metrics-certs\") pod \"maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff\" (UID: \"c107238c-2f65-454b-b5ce-57360fd751a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff"
Sep 17 00:43:58.820556 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:58.820356    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlvq9\" (UniqueName: \"kubernetes.io/projected/c107238c-2f65-454b-b5ce-57360fd751a0-kube-api-access-nlvq9\") pod \"maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff\" (UID: \"c107238c-2f65-454b-b5ce-57360fd751a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff"
Sep 17 00:43:58.820556 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:58.820393    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c107238c-2f65-454b-b5ce-57360fd751a0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff\" (UID: \"c107238c-2f65-454b-b5ce-57360fd751a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff"
Sep 17 00:43:58.820556 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:58.820516    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c107238c-2f65-454b-b5ce-57360fd751a0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff\" (UID: \"c107238c-2f65-454b-b5ce-57360fd751a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff"
Sep 17 00:43:58.921337 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:58.921288    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c107238c-2f65-454b-b5ce-57360fd751a0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff\" (UID: \"c107238c-2f65-454b-b5ce-57360fd751a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff"
Sep 17 00:43:58.921567 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:58.921352    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c107238c-2f65-454b-b5ce-57360fd751a0-metrics-certs\") pod \"maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff\" (UID: \"c107238c-2f65-454b-b5ce-57360fd751a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff"
Sep 17 00:43:58.921567 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:58.921398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlvq9\" (UniqueName: \"kubernetes.io/projected/c107238c-2f65-454b-b5ce-57360fd751a0-kube-api-access-nlvq9\") pod \"maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff\" (UID: \"c107238c-2f65-454b-b5ce-57360fd751a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff"
Sep 17 00:43:58.921567 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:58.921463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c107238c-2f65-454b-b5ce-57360fd751a0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff\" (UID: \"c107238c-2f65-454b-b5ce-57360fd751a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff"
Sep 17 00:43:58.922029 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:58.921996    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c107238c-2f65-454b-b5ce-57360fd751a0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff\" (UID: \"c107238c-2f65-454b-b5ce-57360fd751a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff"
Sep 17 00:43:58.924107 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:58.924078    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c107238c-2f65-454b-b5ce-57360fd751a0-metrics-certs\") pod \"maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff\" (UID: \"c107238c-2f65-454b-b5ce-57360fd751a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff"
Sep 17 00:43:58.924259 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:58.924136    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c107238c-2f65-454b-b5ce-57360fd751a0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff\" (UID: \"c107238c-2f65-454b-b5ce-57360fd751a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff"
Sep 17 00:43:58.930281 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:58.930254    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlvq9\" (UniqueName: \"kubernetes.io/projected/c107238c-2f65-454b-b5ce-57360fd751a0-kube-api-access-nlvq9\") pod \"maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff\" (UID: \"c107238c-2f65-454b-b5ce-57360fd751a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff"
Sep 17 00:43:59.028775 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:59.028681    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff"
Sep 17 00:43:59.307759 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:59.307730    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff"]
Sep 17 00:43:59.310288 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:43:59.310245    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc107238c_2f65_454b_b5ce_57360fd751a0.slice/crio-1e563fedca5f330467d1fa7ba4b480216fe78bf467049685cff2dd7edd8146d5 WatchSource:0}: Error finding container 1e563fedca5f330467d1fa7ba4b480216fe78bf467049685cff2dd7edd8146d5: Status 404 returned error can't find the container with id 1e563fedca5f330467d1fa7ba4b480216fe78bf467049685cff2dd7edd8146d5
Sep 17 00:43:59.311720 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:43:59.311700    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 00:44:00.272369 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:00.272333    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff" event={"ID":"c107238c-2f65-454b-b5ce-57360fd751a0","Type":"ContainerStarted","Data":"3f8dfdef735ef4015dc27893161ddcb9ffd8e0063f199897179339465b956c54"}
Sep 17 00:44:00.272787 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:00.272398    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff" event={"ID":"c107238c-2f65-454b-b5ce-57360fd751a0","Type":"ContainerStarted","Data":"1e563fedca5f330467d1fa7ba4b480216fe78bf467049685cff2dd7edd8146d5"}
Sep 17 00:44:00.272787 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:00.272532    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff"
Sep 17 00:44:00.288075 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:00.288014    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff" podStartSLOduration=1.8992594569999999 podStartE2EDuration="2.287997771s" totalImagesPullingTime="388.738314ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:44:00.287369181 +0000 UTC m=+2053.145862316" watchObservedRunningTime="2026-09-17 00:44:00.287997771 +0000 UTC m=+2053.146490906"
Sep 17 00:44:04.006230 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:04.006189    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:44:06.284508 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:06.284479    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff"
Sep 17 00:44:42.899975 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:42.899933    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"]
Sep 17 00:44:42.903383 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:42.903364    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:44:42.907246 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:42.907220    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-eaba37\"/\"ipp-route-eaba37-kserve-self-signed-certs\""
Sep 17 00:44:42.907246 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:42.907219    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-eaba37\"/\"openshift-service-ca.crt\""
Sep 17 00:44:42.907472 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:42.907317    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-eaba37\"/\"kube-root-ca.crt\""
Sep 17 00:44:42.907528 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:42.907470    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-eaba37\"/\"default-dockercfg-hsgnh\""
Sep 17 00:44:42.913259 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:42.913237    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"]
Sep 17 00:44:42.993236 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:42.993203    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-home\") pod \"ipp-route-eaba37-kserve-588c5f6fdc-fd8qf\" (UID: \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\") " pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:44:42.993396 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:42.993287    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-tls-certs\") pod \"ipp-route-eaba37-kserve-588c5f6fdc-fd8qf\" (UID: \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\") " pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:44:42.993396 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:42.993308    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-dshm\") pod \"ipp-route-eaba37-kserve-588c5f6fdc-fd8qf\" (UID: \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\") " pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:44:42.993396 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:42.993330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-tmp-dir\") pod \"ipp-route-eaba37-kserve-588c5f6fdc-fd8qf\" (UID: \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\") " pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:44:42.993553 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:42.993386    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdkcz\" (UniqueName: \"kubernetes.io/projected/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-kube-api-access-vdkcz\") pod \"ipp-route-eaba37-kserve-588c5f6fdc-fd8qf\" (UID: \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\") " pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:44:42.993553 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:42.993450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-model-cache\") pod \"ipp-route-eaba37-kserve-588c5f6fdc-fd8qf\" (UID: \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\") " pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:44:43.094001 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:43.093956    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-tls-certs\") pod \"ipp-route-eaba37-kserve-588c5f6fdc-fd8qf\" (UID: \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\") " pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:44:43.094001 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:43.093996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-dshm\") pod \"ipp-route-eaba37-kserve-588c5f6fdc-fd8qf\" (UID: \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\") " pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:44:43.094262 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:43.094027    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-tmp-dir\") pod \"ipp-route-eaba37-kserve-588c5f6fdc-fd8qf\" (UID: \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\") " pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:44:43.094262 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:43.094151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdkcz\" (UniqueName: \"kubernetes.io/projected/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-kube-api-access-vdkcz\") pod \"ipp-route-eaba37-kserve-588c5f6fdc-fd8qf\" (UID: \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\") " pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:44:43.094262 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:43.094199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-model-cache\") pod \"ipp-route-eaba37-kserve-588c5f6fdc-fd8qf\" (UID: \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\") " pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:44:43.094456 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:43.094274    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-home\") pod \"ipp-route-eaba37-kserve-588c5f6fdc-fd8qf\" (UID: \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\") " pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:44:43.094524 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:43.094457    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-tmp-dir\") pod \"ipp-route-eaba37-kserve-588c5f6fdc-fd8qf\" (UID: \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\") " pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:44:43.094581 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:43.094560    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-model-cache\") pod \"ipp-route-eaba37-kserve-588c5f6fdc-fd8qf\" (UID: \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\") " pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:44:43.094641 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:43.094619    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-home\") pod \"ipp-route-eaba37-kserve-588c5f6fdc-fd8qf\" (UID: \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\") " pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:44:43.096523 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:43.096501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-dshm\") pod \"ipp-route-eaba37-kserve-588c5f6fdc-fd8qf\" (UID: \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\") " pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:44:43.096657 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:43.096641    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-tls-certs\") pod \"ipp-route-eaba37-kserve-588c5f6fdc-fd8qf\" (UID: \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\") " pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:44:43.101601 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:43.101577    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdkcz\" (UniqueName: \"kubernetes.io/projected/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-kube-api-access-vdkcz\") pod \"ipp-route-eaba37-kserve-588c5f6fdc-fd8qf\" (UID: \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\") " pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:44:43.213805 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:43.213775    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:44:43.487391 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:43.487333    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"]
Sep 17 00:44:43.489719 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:44:43.489690    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a05dcb5_b7d5_4a6d_a919_b43d9eed0c6f.slice/crio-169014606d51dcd3c085d6268593571887ae0e2978444e70362bd67c5c01e06a WatchSource:0}: Error finding container 169014606d51dcd3c085d6268593571887ae0e2978444e70362bd67c5c01e06a: Status 404 returned error can't find the container with id 169014606d51dcd3c085d6268593571887ae0e2978444e70362bd67c5c01e06a
Sep 17 00:44:44.500966 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:44.500925    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf" event={"ID":"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f","Type":"ContainerStarted","Data":"91d2a41cb1eac57eeca86ae17269187fcd1c22ea01933d807bd5d493a5a969ff"}
Sep 17 00:44:44.500966 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:44.500973    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf" event={"ID":"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f","Type":"ContainerStarted","Data":"169014606d51dcd3c085d6268593571887ae0e2978444e70362bd67c5c01e06a"}
Sep 17 00:44:44.517736 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:44.517686    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf" podStartSLOduration=2.5176721029999998 podStartE2EDuration="2.517672103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:44:44.515080098 +0000 UTC m=+2097.373573245" watchObservedRunningTime="2026-09-17 00:44:44.517672103 +0000 UTC m=+2097.376165237"
Sep 17 00:44:47.007715 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:47.007677    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:44:47.738826 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:47.738796    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/ovn-acl-logging/0.log"
Sep 17 00:44:47.761009 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:47.760982    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/ovn-acl-logging/0.log"
Sep 17 00:44:53.214062 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:53.214015    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:44:53.214062 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:53.214071    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:44:53.226665 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:53.226633    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:44:53.561840 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:44:53.561756    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:45:00.149107 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:00.149062    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a8402d-29826765-spkm2"]
Sep 17 00:45:00.152924 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:00.152876    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-47a2db-29826765-xpd46"]
Sep 17 00:45:00.153920 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:00.153867    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a8402d-29826765-spkm2"
Sep 17 00:45:00.156768 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:00.156745    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-47a2db-29826765-xpd46"]
Sep 17 00:45:00.156939 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:00.156837    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-47a2db-29826765-xpd46"
Sep 17 00:45:00.157302 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:00.157265    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jmwpp\""
Sep 17 00:45:00.158435 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:00.158346    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a8402d-29826765-spkm2"]
Sep 17 00:45:00.168012 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:00.167987    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a8402d-29826765-spkm2"
Sep 17 00:45:00.174540 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:00.174518    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-47a2db-29826765-xpd46"
Sep 17 00:45:00.443918 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:00.443883    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a8402d-29826765-spkm2"]
Sep 17 00:45:00.445714 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:45:00.445690    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12faed73_323e_4223_9ecb_67f795f7e7fe.slice/crio-7cf252b8993d54d85812fc83f32a9a08bf411f702a915b61fa8836304fa43c88 WatchSource:0}: Error finding container 7cf252b8993d54d85812fc83f32a9a08bf411f702a915b61fa8836304fa43c88: Status 404 returned error can't find the container with id 7cf252b8993d54d85812fc83f32a9a08bf411f702a915b61fa8836304fa43c88
Sep 17 00:45:00.450590 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:00.450568    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-47a2db-29826765-xpd46"]
Sep 17 00:45:00.452671 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:45:00.452548    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee209c1_56e5_42be_adf3_2aecdf8e98d4.slice/crio-54d7dae2c0379cd6b5e9b8690a27db14132f8b84d9466ebaf452711bca5665b2 WatchSource:0}: Error finding container 54d7dae2c0379cd6b5e9b8690a27db14132f8b84d9466ebaf452711bca5665b2: Status 404 returned error can't find the container with id 54d7dae2c0379cd6b5e9b8690a27db14132f8b84d9466ebaf452711bca5665b2
Sep 17 00:45:00.588238 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:00.588202    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-47a2db-29826765-xpd46" event={"ID":"0ee209c1-56e5-42be-adf3-2aecdf8e98d4","Type":"ContainerStarted","Data":"54d7dae2c0379cd6b5e9b8690a27db14132f8b84d9466ebaf452711bca5665b2"}
Sep 17 00:45:00.589449 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:00.589403    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a8402d-29826765-spkm2" event={"ID":"12faed73-323e-4223-9ecb-67f795f7e7fe","Type":"ContainerStarted","Data":"7cf252b8993d54d85812fc83f32a9a08bf411f702a915b61fa8836304fa43c88"}
Sep 17 00:45:01.595978 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:01.595940    2577 generic.go:356] "Generic (PLEG): container finished" podID="12faed73-323e-4223-9ecb-67f795f7e7fe" containerID="3a276c4de88145a7f01d1d26dbf574aceb75847c93fec7d363b6fad643d5992c" exitCode=0
Sep 17 00:45:01.596386 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:01.596028    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a8402d-29826765-spkm2" event={"ID":"12faed73-323e-4223-9ecb-67f795f7e7fe","Type":"ContainerDied","Data":"3a276c4de88145a7f01d1d26dbf574aceb75847c93fec7d363b6fad643d5992c"}
Sep 17 00:45:01.597570 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:01.597552    2577 generic.go:356] "Generic (PLEG): container finished" podID="0ee209c1-56e5-42be-adf3-2aecdf8e98d4" containerID="63b4a7844c311a5525baebfec99a20348ffc76330cdbc6375b82cd8f52e705b3" exitCode=0
Sep 17 00:45:01.597663 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:01.597602    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-47a2db-29826765-xpd46" event={"ID":"0ee209c1-56e5-42be-adf3-2aecdf8e98d4","Type":"ContainerDied","Data":"63b4a7844c311a5525baebfec99a20348ffc76330cdbc6375b82cd8f52e705b3"}
Sep 17 00:45:02.735782 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:02.735756    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a8402d-29826765-spkm2"
Sep 17 00:45:02.766131 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:02.766101    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-47a2db-29826765-xpd46"
Sep 17 00:45:03.608906 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:03.608872    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a8402d-29826765-spkm2"
Sep 17 00:45:03.608906 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:03.608893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a8402d-29826765-spkm2" event={"ID":"12faed73-323e-4223-9ecb-67f795f7e7fe","Type":"ContainerDied","Data":"7cf252b8993d54d85812fc83f32a9a08bf411f702a915b61fa8836304fa43c88"}
Sep 17 00:45:03.609196 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:03.608946    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cf252b8993d54d85812fc83f32a9a08bf411f702a915b61fa8836304fa43c88"
Sep 17 00:45:03.610297 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:03.610272    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-47a2db-29826765-xpd46" event={"ID":"0ee209c1-56e5-42be-adf3-2aecdf8e98d4","Type":"ContainerDied","Data":"54d7dae2c0379cd6b5e9b8690a27db14132f8b84d9466ebaf452711bca5665b2"}
Sep 17 00:45:03.610466 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:03.610307    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-47a2db-29826765-xpd46"
Sep 17 00:45:03.610466 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:03.610313    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54d7dae2c0379cd6b5e9b8690a27db14132f8b84d9466ebaf452711bca5665b2"
Sep 17 00:45:07.504124 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:07.504088    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:45:20.734961 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:20.734924    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g"]
Sep 17 00:45:20.735438 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:20.735245    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12faed73-323e-4223-9ecb-67f795f7e7fe" containerName="cleanup"
Sep 17 00:45:20.735438 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:20.735256    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="12faed73-323e-4223-9ecb-67f795f7e7fe" containerName="cleanup"
Sep 17 00:45:20.735438 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:20.735266    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ee209c1-56e5-42be-adf3-2aecdf8e98d4" containerName="cleanup"
Sep 17 00:45:20.735438 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:20.735271    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ee209c1-56e5-42be-adf3-2aecdf8e98d4" containerName="cleanup"
Sep 17 00:45:20.735438 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:20.735334    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ee209c1-56e5-42be-adf3-2aecdf8e98d4" containerName="cleanup"
Sep 17 00:45:20.735438 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:20.735345    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="12faed73-323e-4223-9ecb-67f795f7e7fe" containerName="cleanup"
Sep 17 00:45:20.738638 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:20.738615    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g"
Sep 17 00:45:20.741126 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:20.741105    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-cfebd7-serving-cert\""
Sep 17 00:45:20.749735 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:20.749707    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g"]
Sep 17 00:45:20.802074 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:20.802043    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g\" (UID: \"cd9ee60e-8610-44b7-b2a1-70c4ad7329a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g"
Sep 17 00:45:20.802236 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:20.802088    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6dhc\" (UniqueName: \"kubernetes.io/projected/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3-kube-api-access-j6dhc\") pod \"maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g\" (UID: \"cd9ee60e-8610-44b7-b2a1-70c4ad7329a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g"
Sep 17 00:45:20.802236 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:20.802107    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g\" (UID: \"cd9ee60e-8610-44b7-b2a1-70c4ad7329a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g"
Sep 17 00:45:20.802236 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:20.802133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g\" (UID: \"cd9ee60e-8610-44b7-b2a1-70c4ad7329a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g"
Sep 17 00:45:20.902833 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:20.902797    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g\" (UID: \"cd9ee60e-8610-44b7-b2a1-70c4ad7329a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g"
Sep 17 00:45:20.902833 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:20.902845    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6dhc\" (UniqueName: \"kubernetes.io/projected/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3-kube-api-access-j6dhc\") pod \"maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g\" (UID: \"cd9ee60e-8610-44b7-b2a1-70c4ad7329a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g"
Sep 17 00:45:20.903083 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:20.902862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g\" (UID: \"cd9ee60e-8610-44b7-b2a1-70c4ad7329a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g"
Sep 17 00:45:20.903083 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:20.902883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g\" (UID: \"cd9ee60e-8610-44b7-b2a1-70c4ad7329a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g"
Sep 17 00:45:20.903658 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:20.903633    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g\" (UID: \"cd9ee60e-8610-44b7-b2a1-70c4ad7329a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g"
Sep 17 00:45:20.905246 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:20.905226    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g\" (UID: \"cd9ee60e-8610-44b7-b2a1-70c4ad7329a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g"
Sep 17 00:45:20.905369 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:20.905313    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g\" (UID: \"cd9ee60e-8610-44b7-b2a1-70c4ad7329a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g"
Sep 17 00:45:20.911515 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:20.911486    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6dhc\" (UniqueName: \"kubernetes.io/projected/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3-kube-api-access-j6dhc\") pod \"maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g\" (UID: \"cd9ee60e-8610-44b7-b2a1-70c4ad7329a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g"
Sep 17 00:45:21.052116 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:21.052025    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g"
Sep 17 00:45:21.325819 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:21.325789    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g"]
Sep 17 00:45:21.326922 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:45:21.326894    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9ee60e_8610_44b7_b2a1_70c4ad7329a3.slice/crio-a6c3a74093576ec0c33daf15c361172c7f7a1fdaa9d592219908ad12d80dd82b WatchSource:0}: Error finding container a6c3a74093576ec0c33daf15c361172c7f7a1fdaa9d592219908ad12d80dd82b: Status 404 returned error can't find the container with id a6c3a74093576ec0c33daf15c361172c7f7a1fdaa9d592219908ad12d80dd82b
Sep 17 00:45:21.708970 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:21.708934    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g" event={"ID":"cd9ee60e-8610-44b7-b2a1-70c4ad7329a3","Type":"ContainerStarted","Data":"a6c3a74093576ec0c33daf15c361172c7f7a1fdaa9d592219908ad12d80dd82b"}
Sep 17 00:45:22.716063 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:22.716028    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g" event={"ID":"cd9ee60e-8610-44b7-b2a1-70c4ad7329a3","Type":"ContainerStarted","Data":"c7c9b8b2ec7d177258d69f9442894ecee2e5a0dfd7096797565f28f62ecc98d9"}
Sep 17 00:45:22.716541 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:22.716088    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g"
Sep 17 00:45:22.732994 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:22.732949    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g" podStartSLOduration=2.265268485 podStartE2EDuration="2.732934222s" totalImagesPullingTime="467.665737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:45:22.731576574 +0000 UTC m=+2135.590069708" watchObservedRunningTime="2026-09-17 00:45:22.732934222 +0000 UTC m=+2135.591427356"
Sep 17 00:45:25.209310 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:25.209280    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:45:28.729271 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:28.729241    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g"
Sep 17 00:45:49.754088 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:49.753884    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g"]
Sep 17 00:45:49.756630 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:49.754177    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g" podUID="cd9ee60e-8610-44b7-b2a1-70c4ad7329a3" containerName="maas-api" containerID="cri-o://c7c9b8b2ec7d177258d69f9442894ecee2e5a0dfd7096797565f28f62ecc98d9" gracePeriod=30
Sep 17 00:45:49.999238 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:49.999209    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g"
Sep 17 00:45:50.047378 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:50.047300    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3-openshift-service-ca\") pod \"cd9ee60e-8610-44b7-b2a1-70c4ad7329a3\" (UID: \"cd9ee60e-8610-44b7-b2a1-70c4ad7329a3\") "
Sep 17 00:45:50.047546 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:50.047385    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6dhc\" (UniqueName: \"kubernetes.io/projected/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3-kube-api-access-j6dhc\") pod \"cd9ee60e-8610-44b7-b2a1-70c4ad7329a3\" (UID: \"cd9ee60e-8610-44b7-b2a1-70c4ad7329a3\") "
Sep 17 00:45:50.047546 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:50.047424    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3-metrics-certs\") pod \"cd9ee60e-8610-44b7-b2a1-70c4ad7329a3\" (UID: \"cd9ee60e-8610-44b7-b2a1-70c4ad7329a3\") "
Sep 17 00:45:50.047546 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:50.047451    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3-maas-api-tls\") pod \"cd9ee60e-8610-44b7-b2a1-70c4ad7329a3\" (UID: \"cd9ee60e-8610-44b7-b2a1-70c4ad7329a3\") "
Sep 17 00:45:50.047774 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:50.047744    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cd9ee60e-8610-44b7-b2a1-70c4ad7329a3" (UID: "cd9ee60e-8610-44b7-b2a1-70c4ad7329a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:45:50.049538 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:50.049510    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cd9ee60e-8610-44b7-b2a1-70c4ad7329a3" (UID: "cd9ee60e-8610-44b7-b2a1-70c4ad7329a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:45:50.049674 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:50.049653    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3-kube-api-access-j6dhc" (OuterVolumeSpecName: "kube-api-access-j6dhc") pod "cd9ee60e-8610-44b7-b2a1-70c4ad7329a3" (UID: "cd9ee60e-8610-44b7-b2a1-70c4ad7329a3"). InnerVolumeSpecName "kube-api-access-j6dhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:45:50.049865 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:50.049845    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cd9ee60e-8610-44b7-b2a1-70c4ad7329a3" (UID: "cd9ee60e-8610-44b7-b2a1-70c4ad7329a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:45:50.148624 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:50.148583    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3-openshift-service-ca\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:45:50.148624 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:50.148613    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6dhc\" (UniqueName: \"kubernetes.io/projected/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3-kube-api-access-j6dhc\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:45:50.148624 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:50.148624    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3-metrics-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:45:50.148624 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:50.148634    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3-maas-api-tls\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:45:50.869197 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:50.869157    2577 generic.go:356] "Generic (PLEG): container finished" podID="cd9ee60e-8610-44b7-b2a1-70c4ad7329a3" containerID="c7c9b8b2ec7d177258d69f9442894ecee2e5a0dfd7096797565f28f62ecc98d9" exitCode=0
Sep 17 00:45:50.869631 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:50.869203    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g"
Sep 17 00:45:50.869631 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:50.869232    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g" event={"ID":"cd9ee60e-8610-44b7-b2a1-70c4ad7329a3","Type":"ContainerDied","Data":"c7c9b8b2ec7d177258d69f9442894ecee2e5a0dfd7096797565f28f62ecc98d9"}
Sep 17 00:45:50.869631 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:50.869281    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g" event={"ID":"cd9ee60e-8610-44b7-b2a1-70c4ad7329a3","Type":"ContainerDied","Data":"a6c3a74093576ec0c33daf15c361172c7f7a1fdaa9d592219908ad12d80dd82b"}
Sep 17 00:45:50.869631 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:50.869306    2577 scope.go:117] "RemoveContainer" containerID="c7c9b8b2ec7d177258d69f9442894ecee2e5a0dfd7096797565f28f62ecc98d9"
Sep 17 00:45:50.878393 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:50.878376    2577 scope.go:117] "RemoveContainer" containerID="c7c9b8b2ec7d177258d69f9442894ecee2e5a0dfd7096797565f28f62ecc98d9"
Sep 17 00:45:50.878676 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:45:50.878657    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c9b8b2ec7d177258d69f9442894ecee2e5a0dfd7096797565f28f62ecc98d9\": container with ID starting with c7c9b8b2ec7d177258d69f9442894ecee2e5a0dfd7096797565f28f62ecc98d9 not found: ID does not exist" containerID="c7c9b8b2ec7d177258d69f9442894ecee2e5a0dfd7096797565f28f62ecc98d9"
Sep 17 00:45:50.878732 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:50.878685    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c9b8b2ec7d177258d69f9442894ecee2e5a0dfd7096797565f28f62ecc98d9"} err="failed to get container status \"c7c9b8b2ec7d177258d69f9442894ecee2e5a0dfd7096797565f28f62ecc98d9\": rpc error: code = NotFound desc = could not find container \"c7c9b8b2ec7d177258d69f9442894ecee2e5a0dfd7096797565f28f62ecc98d9\": container with ID starting with c7c9b8b2ec7d177258d69f9442894ecee2e5a0dfd7096797565f28f62ecc98d9 not found: ID does not exist"
Sep 17 00:45:50.896127 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:50.896101    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g"]
Sep 17 00:45:50.898991 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:50.898972    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cfebd7-6c765467c8-tjg2g"]
Sep 17 00:45:51.608201 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:51.608169    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd9ee60e-8610-44b7-b2a1-70c4ad7329a3" path="/var/lib/kubelet/pods/cd9ee60e-8610-44b7-b2a1-70c4ad7329a3/volumes"
Sep 17 00:45:54.298033 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:45:54.297995    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:46:12.897082 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:12.897044    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:46:20.993273 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:20.993232    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"]
Sep 17 00:46:24.701847 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:24.701806    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:46:36.012676 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:36.012633    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf" podUID="2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f" containerName="main" containerID="cri-o://91d2a41cb1eac57eeca86ae17269187fcd1c22ea01933d807bd5d493a5a969ff" gracePeriod=45
Sep 17 00:46:36.261367 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:36.261343    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:46:36.352098 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:36.352015    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-home\") pod \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\" (UID: \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\") "
Sep 17 00:46:36.352098 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:36.352063    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-dshm\") pod \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\" (UID: \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\") "
Sep 17 00:46:36.352313 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:36.352140    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-tls-certs\") pod \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\" (UID: \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\") "
Sep 17 00:46:36.352313 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:36.352163    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-tmp-dir\") pod \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\" (UID: \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\") "
Sep 17 00:46:36.352313 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:36.352183    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-model-cache\") pod \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\" (UID: \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\") "
Sep 17 00:46:36.352313 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:36.352201    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdkcz\" (UniqueName: \"kubernetes.io/projected/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-kube-api-access-vdkcz\") pod \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\" (UID: \"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f\") "
Sep 17 00:46:36.352563 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:36.352329    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-home" (OuterVolumeSpecName: "home") pod "2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f" (UID: "2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:46:36.352563 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:36.352475    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-home\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:46:36.352563 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:36.352496    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f" (UID: "2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:46:36.352780 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:36.352757    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f" (UID: "2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:46:36.354263 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:36.354241    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-dshm" (OuterVolumeSpecName: "dshm") pod "2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f" (UID: "2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:46:36.354332 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:36.354283    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f" (UID: "2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:46:36.354426 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:36.354391    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-kube-api-access-vdkcz" (OuterVolumeSpecName: "kube-api-access-vdkcz") pod "2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f" (UID: "2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f"). InnerVolumeSpecName "kube-api-access-vdkcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:46:36.453114 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:36.453072    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-tls-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:46:36.453114 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:36.453109    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-tmp-dir\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:46:36.453114 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:36.453119    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-model-cache\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:46:36.453114 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:36.453127    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdkcz\" (UniqueName: \"kubernetes.io/projected/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-kube-api-access-vdkcz\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:46:36.453349 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:36.453136    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f-dshm\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:46:37.122147 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:37.122116    2577 generic.go:356] "Generic (PLEG): container finished" podID="2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f" containerID="91d2a41cb1eac57eeca86ae17269187fcd1c22ea01933d807bd5d493a5a969ff" exitCode=0
Sep 17 00:46:37.122596 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:37.122156    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"
Sep 17 00:46:37.122596 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:37.122183    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf" event={"ID":"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f","Type":"ContainerDied","Data":"91d2a41cb1eac57eeca86ae17269187fcd1c22ea01933d807bd5d493a5a969ff"}
Sep 17 00:46:37.122596 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:37.122214    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf" event={"ID":"2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f","Type":"ContainerDied","Data":"169014606d51dcd3c085d6268593571887ae0e2978444e70362bd67c5c01e06a"}
Sep 17 00:46:37.122596 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:37.122232    2577 scope.go:117] "RemoveContainer" containerID="91d2a41cb1eac57eeca86ae17269187fcd1c22ea01933d807bd5d493a5a969ff"
Sep 17 00:46:37.132250 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:37.132231    2577 scope.go:117] "RemoveContainer" containerID="91d2a41cb1eac57eeca86ae17269187fcd1c22ea01933d807bd5d493a5a969ff"
Sep 17 00:46:37.132549 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:46:37.132528    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d2a41cb1eac57eeca86ae17269187fcd1c22ea01933d807bd5d493a5a969ff\": container with ID starting with 91d2a41cb1eac57eeca86ae17269187fcd1c22ea01933d807bd5d493a5a969ff not found: ID does not exist" containerID="91d2a41cb1eac57eeca86ae17269187fcd1c22ea01933d807bd5d493a5a969ff"
Sep 17 00:46:37.132626 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:37.132557    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d2a41cb1eac57eeca86ae17269187fcd1c22ea01933d807bd5d493a5a969ff"} err="failed to get container status \"91d2a41cb1eac57eeca86ae17269187fcd1c22ea01933d807bd5d493a5a969ff\": rpc error: code = NotFound desc = could not find container \"91d2a41cb1eac57eeca86ae17269187fcd1c22ea01933d807bd5d493a5a969ff\": container with ID starting with 91d2a41cb1eac57eeca86ae17269187fcd1c22ea01933d807bd5d493a5a969ff not found: ID does not exist"
Sep 17 00:46:37.143088 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:37.143062    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"]
Sep 17 00:46:37.146444 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:37.146399    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-eaba37/ipp-route-eaba37-kserve-588c5f6fdc-fd8qf"]
Sep 17 00:46:37.608696 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:46:37.608661    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f" path="/var/lib/kubelet/pods/2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f/volumes"
Sep 17 00:47:14.608190 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:14.607971    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff"]
Sep 17 00:47:14.608796 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:14.608260    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff" podUID="c107238c-2f65-454b-b5ce-57360fd751a0" containerName="maas-api" containerID="cri-o://3f8dfdef735ef4015dc27893161ddcb9ffd8e0063f199897179339465b956c54" gracePeriod=30
Sep 17 00:47:14.640862 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:14.640828    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a8402d-29826765-spkm2"]
Sep 17 00:47:14.643069 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:14.643041    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a8402d-29826765-spkm2"]
Sep 17 00:47:14.857166 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:14.857135    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff"
Sep 17 00:47:14.979887 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:14.979857    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c107238c-2f65-454b-b5ce-57360fd751a0-openshift-service-ca\") pod \"c107238c-2f65-454b-b5ce-57360fd751a0\" (UID: \"c107238c-2f65-454b-b5ce-57360fd751a0\") "
Sep 17 00:47:14.980077 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:14.979922    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c107238c-2f65-454b-b5ce-57360fd751a0-maas-api-tls\") pod \"c107238c-2f65-454b-b5ce-57360fd751a0\" (UID: \"c107238c-2f65-454b-b5ce-57360fd751a0\") "
Sep 17 00:47:14.980077 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:14.979966    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlvq9\" (UniqueName: \"kubernetes.io/projected/c107238c-2f65-454b-b5ce-57360fd751a0-kube-api-access-nlvq9\") pod \"c107238c-2f65-454b-b5ce-57360fd751a0\" (UID: \"c107238c-2f65-454b-b5ce-57360fd751a0\") "
Sep 17 00:47:14.980077 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:14.979981    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c107238c-2f65-454b-b5ce-57360fd751a0-metrics-certs\") pod \"c107238c-2f65-454b-b5ce-57360fd751a0\" (UID: \"c107238c-2f65-454b-b5ce-57360fd751a0\") "
Sep 17 00:47:14.980346 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:14.980308    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c107238c-2f65-454b-b5ce-57360fd751a0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c107238c-2f65-454b-b5ce-57360fd751a0" (UID: "c107238c-2f65-454b-b5ce-57360fd751a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:47:14.982138 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:14.982102    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c107238c-2f65-454b-b5ce-57360fd751a0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c107238c-2f65-454b-b5ce-57360fd751a0" (UID: "c107238c-2f65-454b-b5ce-57360fd751a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:47:14.982138 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:14.982128    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c107238c-2f65-454b-b5ce-57360fd751a0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c107238c-2f65-454b-b5ce-57360fd751a0" (UID: "c107238c-2f65-454b-b5ce-57360fd751a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:47:14.982287 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:14.982215    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c107238c-2f65-454b-b5ce-57360fd751a0-kube-api-access-nlvq9" (OuterVolumeSpecName: "kube-api-access-nlvq9") pod "c107238c-2f65-454b-b5ce-57360fd751a0" (UID: "c107238c-2f65-454b-b5ce-57360fd751a0"). InnerVolumeSpecName "kube-api-access-nlvq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:47:15.081472 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:15.081397    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c107238c-2f65-454b-b5ce-57360fd751a0-openshift-service-ca\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:47:15.081472 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:15.081468    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c107238c-2f65-454b-b5ce-57360fd751a0-maas-api-tls\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:47:15.081695 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:15.081480    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlvq9\" (UniqueName: \"kubernetes.io/projected/c107238c-2f65-454b-b5ce-57360fd751a0-kube-api-access-nlvq9\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:47:15.081695 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:15.081523    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c107238c-2f65-454b-b5ce-57360fd751a0-metrics-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:47:15.326982 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:15.326884    2577 generic.go:356] "Generic (PLEG): container finished" podID="c107238c-2f65-454b-b5ce-57360fd751a0" containerID="3f8dfdef735ef4015dc27893161ddcb9ffd8e0063f199897179339465b956c54" exitCode=0
Sep 17 00:47:15.326982 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:15.326939    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff"
Sep 17 00:47:15.327166 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:15.326974    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff" event={"ID":"c107238c-2f65-454b-b5ce-57360fd751a0","Type":"ContainerDied","Data":"3f8dfdef735ef4015dc27893161ddcb9ffd8e0063f199897179339465b956c54"}
Sep 17 00:47:15.327166 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:15.327053    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff" event={"ID":"c107238c-2f65-454b-b5ce-57360fd751a0","Type":"ContainerDied","Data":"1e563fedca5f330467d1fa7ba4b480216fe78bf467049685cff2dd7edd8146d5"}
Sep 17 00:47:15.327166 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:15.327078    2577 scope.go:117] "RemoveContainer" containerID="3f8dfdef735ef4015dc27893161ddcb9ffd8e0063f199897179339465b956c54"
Sep 17 00:47:15.336734 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:15.336711    2577 scope.go:117] "RemoveContainer" containerID="3f8dfdef735ef4015dc27893161ddcb9ffd8e0063f199897179339465b956c54"
Sep 17 00:47:15.337064 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:47:15.337045    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f8dfdef735ef4015dc27893161ddcb9ffd8e0063f199897179339465b956c54\": container with ID starting with 3f8dfdef735ef4015dc27893161ddcb9ffd8e0063f199897179339465b956c54 not found: ID does not exist" containerID="3f8dfdef735ef4015dc27893161ddcb9ffd8e0063f199897179339465b956c54"
Sep 17 00:47:15.337119 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:15.337074    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8dfdef735ef4015dc27893161ddcb9ffd8e0063f199897179339465b956c54"} err="failed to get container status \"3f8dfdef735ef4015dc27893161ddcb9ffd8e0063f199897179339465b956c54\": rpc error: code = NotFound desc = could not find container \"3f8dfdef735ef4015dc27893161ddcb9ffd8e0063f199897179339465b956c54\": container with ID starting with 3f8dfdef735ef4015dc27893161ddcb9ffd8e0063f199897179339465b956c54 not found: ID does not exist"
Sep 17 00:47:15.350290 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:15.350260    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff"]
Sep 17 00:47:15.353554 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:15.353527    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a8402d-86c4877464-dl7ff"]
Sep 17 00:47:15.608141 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:15.608048    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12faed73-323e-4223-9ecb-67f795f7e7fe" path="/var/lib/kubelet/pods/12faed73-323e-4223-9ecb-67f795f7e7fe/volumes"
Sep 17 00:47:15.608538 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:15.608519    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c107238c-2f65-454b-b5ce-57360fd751a0" path="/var/lib/kubelet/pods/c107238c-2f65-454b-b5ce-57360fd751a0/volumes"
Sep 17 00:47:18.195568 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:18.195538    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:47:31.995588 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:31.995547    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"]
Sep 17 00:47:31.996054 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:31.995807    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v" podUID="cae07c52-98a2-4244-9d3e-05dd95f8c86d" containerName="maas-api" containerID="cri-o://74aa15f1e0985a078e34d09465c202fedea086083c5177c7217a08b862308d80" gracePeriod=30
Sep 17 00:47:32.029543 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:32.029510    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-47a2db-29826765-xpd46"]
Sep 17 00:47:32.031439 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:32.031401    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-47a2db-29826765-xpd46"]
Sep 17 00:47:32.249823 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:32.249745    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"
Sep 17 00:47:32.405717 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:32.405669    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w2pw\" (UniqueName: \"kubernetes.io/projected/cae07c52-98a2-4244-9d3e-05dd95f8c86d-kube-api-access-4w2pw\") pod \"cae07c52-98a2-4244-9d3e-05dd95f8c86d\" (UID: \"cae07c52-98a2-4244-9d3e-05dd95f8c86d\") "
Sep 17 00:47:32.405717 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:32.405719    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae07c52-98a2-4244-9d3e-05dd95f8c86d-maas-api-tls\") pod \"cae07c52-98a2-4244-9d3e-05dd95f8c86d\" (UID: \"cae07c52-98a2-4244-9d3e-05dd95f8c86d\") "
Sep 17 00:47:32.405981 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:32.405756    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae07c52-98a2-4244-9d3e-05dd95f8c86d-openshift-service-ca\") pod \"cae07c52-98a2-4244-9d3e-05dd95f8c86d\" (UID: \"cae07c52-98a2-4244-9d3e-05dd95f8c86d\") "
Sep 17 00:47:32.405981 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:32.405773    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae07c52-98a2-4244-9d3e-05dd95f8c86d-metrics-certs\") pod \"cae07c52-98a2-4244-9d3e-05dd95f8c86d\" (UID: \"cae07c52-98a2-4244-9d3e-05dd95f8c86d\") "
Sep 17 00:47:32.406138 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:32.406110    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cae07c52-98a2-4244-9d3e-05dd95f8c86d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cae07c52-98a2-4244-9d3e-05dd95f8c86d" (UID: "cae07c52-98a2-4244-9d3e-05dd95f8c86d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:47:32.407910 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:32.407884    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cae07c52-98a2-4244-9d3e-05dd95f8c86d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cae07c52-98a2-4244-9d3e-05dd95f8c86d" (UID: "cae07c52-98a2-4244-9d3e-05dd95f8c86d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:47:32.408014 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:32.407908    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cae07c52-98a2-4244-9d3e-05dd95f8c86d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cae07c52-98a2-4244-9d3e-05dd95f8c86d" (UID: "cae07c52-98a2-4244-9d3e-05dd95f8c86d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:47:32.408014 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:32.407905    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae07c52-98a2-4244-9d3e-05dd95f8c86d-kube-api-access-4w2pw" (OuterVolumeSpecName: "kube-api-access-4w2pw") pod "cae07c52-98a2-4244-9d3e-05dd95f8c86d" (UID: "cae07c52-98a2-4244-9d3e-05dd95f8c86d"). InnerVolumeSpecName "kube-api-access-4w2pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:47:32.422950 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:32.422919    2577 generic.go:356] "Generic (PLEG): container finished" podID="cae07c52-98a2-4244-9d3e-05dd95f8c86d" containerID="74aa15f1e0985a078e34d09465c202fedea086083c5177c7217a08b862308d80" exitCode=0
Sep 17 00:47:32.423087 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:32.422963    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v" event={"ID":"cae07c52-98a2-4244-9d3e-05dd95f8c86d","Type":"ContainerDied","Data":"74aa15f1e0985a078e34d09465c202fedea086083c5177c7217a08b862308d80"}
Sep 17 00:47:32.423087 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:32.422978    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"
Sep 17 00:47:32.423087 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:32.423017    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v" event={"ID":"cae07c52-98a2-4244-9d3e-05dd95f8c86d","Type":"ContainerDied","Data":"7ed827e8f08085c1701b1f7b85f0028caf2c3159b522781305c0de4b5edf1768"}
Sep 17 00:47:32.423087 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:32.423046    2577 scope.go:117] "RemoveContainer" containerID="74aa15f1e0985a078e34d09465c202fedea086083c5177c7217a08b862308d80"
Sep 17 00:47:32.435025 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:32.435004    2577 scope.go:117] "RemoveContainer" containerID="74aa15f1e0985a078e34d09465c202fedea086083c5177c7217a08b862308d80"
Sep 17 00:47:32.435341 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:47:32.435324    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74aa15f1e0985a078e34d09465c202fedea086083c5177c7217a08b862308d80\": container with ID starting with 74aa15f1e0985a078e34d09465c202fedea086083c5177c7217a08b862308d80 not found: ID does not exist" containerID="74aa15f1e0985a078e34d09465c202fedea086083c5177c7217a08b862308d80"
Sep 17 00:47:32.435392 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:32.435349    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74aa15f1e0985a078e34d09465c202fedea086083c5177c7217a08b862308d80"} err="failed to get container status \"74aa15f1e0985a078e34d09465c202fedea086083c5177c7217a08b862308d80\": rpc error: code = NotFound desc = could not find container \"74aa15f1e0985a078e34d09465c202fedea086083c5177c7217a08b862308d80\": container with ID starting with 74aa15f1e0985a078e34d09465c202fedea086083c5177c7217a08b862308d80 not found: ID does not exist"
Sep 17 00:47:32.449677 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:32.449640    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"]
Sep 17 00:47:32.452788 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:32.452762    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-47a2db-57b47bcb57-qrt5v"]
Sep 17 00:47:32.506476 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:32.506377    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae07c52-98a2-4244-9d3e-05dd95f8c86d-openshift-service-ca\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:47:32.506476 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:32.506440    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae07c52-98a2-4244-9d3e-05dd95f8c86d-metrics-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:47:32.506476 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:32.506453    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w2pw\" (UniqueName: \"kubernetes.io/projected/cae07c52-98a2-4244-9d3e-05dd95f8c86d-kube-api-access-4w2pw\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:47:32.506476 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:32.506462    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae07c52-98a2-4244-9d3e-05dd95f8c86d-maas-api-tls\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:47:33.608246 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:33.608206    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ee209c1-56e5-42be-adf3-2aecdf8e98d4" path="/var/lib/kubelet/pods/0ee209c1-56e5-42be-adf3-2aecdf8e98d4/volumes"
Sep 17 00:47:33.608637 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:33.608597    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cae07c52-98a2-4244-9d3e-05dd95f8c86d" path="/var/lib/kubelet/pods/cae07c52-98a2-4244-9d3e-05dd95f8c86d/volumes"
Sep 17 00:47:35.207286 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:35.207249    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:47:38.950818 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:38.950776    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-07f9f2-nm778"]
Sep 17 00:47:38.951301 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:38.951231    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f" containerName="main"
Sep 17 00:47:38.951301 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:38.951251    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f" containerName="main"
Sep 17 00:47:38.951301 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:38.951264    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c107238c-2f65-454b-b5ce-57360fd751a0" containerName="maas-api"
Sep 17 00:47:38.951301 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:38.951274    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c107238c-2f65-454b-b5ce-57360fd751a0" containerName="maas-api"
Sep 17 00:47:38.951301 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:38.951291    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd9ee60e-8610-44b7-b2a1-70c4ad7329a3" containerName="maas-api"
Sep 17 00:47:38.951301 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:38.951300    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd9ee60e-8610-44b7-b2a1-70c4ad7329a3" containerName="maas-api"
Sep 17 00:47:38.951645 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:38.951330    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cae07c52-98a2-4244-9d3e-05dd95f8c86d" containerName="maas-api"
Sep 17 00:47:38.951645 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:38.951339    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cae07c52-98a2-4244-9d3e-05dd95f8c86d" containerName="maas-api"
Sep 17 00:47:38.951645 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:38.951462    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a05dcb5-b7d5-4a6d-a919-b43d9eed0c6f" containerName="main"
Sep 17 00:47:38.951645 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:38.951478    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cae07c52-98a2-4244-9d3e-05dd95f8c86d" containerName="maas-api"
Sep 17 00:47:38.951645 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:38.951490    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c107238c-2f65-454b-b5ce-57360fd751a0" containerName="maas-api"
Sep 17 00:47:38.951645 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:38.951500    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd9ee60e-8610-44b7-b2a1-70c4ad7329a3" containerName="maas-api"
Sep 17 00:47:38.955949 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:38.955921    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-07f9f2-nm778"
Sep 17 00:47:38.962891 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:38.962863    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-07f9f2-nm778"]
Sep 17 00:47:38.963404 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:38.963384    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 00:47:38.963722 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:38.963488    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jmwpp\""
Sep 17 00:47:39.064759 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:39.064717    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a28e2c-2d11-4521-aad9-c6a5846ea441-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-07f9f2-nm778\" (UID: \"98a28e2c-2d11-4521-aad9-c6a5846ea441\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-07f9f2-nm778"
Sep 17 00:47:39.165727 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:39.165686    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a28e2c-2d11-4521-aad9-c6a5846ea441-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-07f9f2-nm778\" (UID: \"98a28e2c-2d11-4521-aad9-c6a5846ea441\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-07f9f2-nm778"
Sep 17 00:47:39.166325 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:39.166304    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a28e2c-2d11-4521-aad9-c6a5846ea441-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-07f9f2-nm778\" (UID: \"98a28e2c-2d11-4521-aad9-c6a5846ea441\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-07f9f2-nm778"
Sep 17 00:47:39.273297 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:39.273206    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-07f9f2-nm778"
Sep 17 00:47:39.542194 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:39.542164    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-07f9f2-nm778"]
Sep 17 00:47:39.544079 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:47:39.544051    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a28e2c_2d11_4521_aad9_c6a5846ea441.slice/crio-2b367dac4f14ff5673a129cef4f9af712c2ff21ac66779364558dd337243cbba WatchSource:0}: Error finding container 2b367dac4f14ff5673a129cef4f9af712c2ff21ac66779364558dd337243cbba: Status 404 returned error can't find the container with id 2b367dac4f14ff5673a129cef4f9af712c2ff21ac66779364558dd337243cbba
Sep 17 00:47:40.467982 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:40.467945    2577 generic.go:356] "Generic (PLEG): container finished" podID="98a28e2c-2d11-4521-aad9-c6a5846ea441" containerID="14f2caa1d1c949943d0fce807155bbc87483ada22dd78f8b6839cfd307bacb8b" exitCode=0
Sep 17 00:47:40.468374 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:40.468003    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-07f9f2-nm778" event={"ID":"98a28e2c-2d11-4521-aad9-c6a5846ea441","Type":"ContainerDied","Data":"14f2caa1d1c949943d0fce807155bbc87483ada22dd78f8b6839cfd307bacb8b"}
Sep 17 00:47:40.468374 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:40.468049    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-07f9f2-nm778" event={"ID":"98a28e2c-2d11-4521-aad9-c6a5846ea441","Type":"ContainerStarted","Data":"2b367dac4f14ff5673a129cef4f9af712c2ff21ac66779364558dd337243cbba"}
Sep 17 00:47:41.600320 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:41.600293    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-07f9f2-nm778"
Sep 17 00:47:41.686571 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:41.686540    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a28e2c-2d11-4521-aad9-c6a5846ea441-maas-api-service-ca\") pod \"98a28e2c-2d11-4521-aad9-c6a5846ea441\" (UID: \"98a28e2c-2d11-4521-aad9-c6a5846ea441\") "
Sep 17 00:47:41.686889 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:41.686867    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a28e2c-2d11-4521-aad9-c6a5846ea441-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "98a28e2c-2d11-4521-aad9-c6a5846ea441" (UID: "98a28e2c-2d11-4521-aad9-c6a5846ea441"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:47:41.787081 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:41.786997    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a28e2c-2d11-4521-aad9-c6a5846ea441-maas-api-service-ca\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:47:42.479271 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:42.479241    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-07f9f2-nm778"
Sep 17 00:47:42.479460 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:42.479239    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-07f9f2-nm778" event={"ID":"98a28e2c-2d11-4521-aad9-c6a5846ea441","Type":"ContainerDied","Data":"2b367dac4f14ff5673a129cef4f9af712c2ff21ac66779364558dd337243cbba"}
Sep 17 00:47:42.479460 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:42.479363    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b367dac4f14ff5673a129cef4f9af712c2ff21ac66779364558dd337243cbba"
Sep 17 00:47:49.767661 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:49.767624    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-07f9f2-nm778"]
Sep 17 00:47:49.770139 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:49.770112    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-07f9f2-nm778"]
Sep 17 00:47:51.606703 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:51.606667    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98a28e2c-2d11-4521-aad9-c6a5846ea441" path="/var/lib/kubelet/pods/98a28e2c-2d11-4521-aad9-c6a5846ea441/volumes"
Sep 17 00:47:52.000998 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:47:52.000963    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:48:00.595376 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:48:00.595340    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:49:20.895030 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:49:20.894994    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:49:46.002040 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:49:46.002003    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:49:47.774679 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:49:47.774648    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/ovn-acl-logging/0.log"
Sep 17 00:49:47.801846 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:49:47.801818    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/ovn-acl-logging/0.log"
Sep 17 00:49:54.396514 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:49:54.396469    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:50:02.097866 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:02.097823    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:50:24.495598 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:24.495518    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:50:29.299623 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:29.299587    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:50:34.162178 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:34.162139    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg"]
Sep 17 00:50:34.162728 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:34.162334    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg" podUID="b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969" containerName="manager" containerID="cri-o://ee7032d0f71608edb5109a82bab6c54e8e3fe94be0188600bfef1b87c8a63b14" gracePeriod=10
Sep 17 00:50:35.322097 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:35.322066    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg"
Sep 17 00:50:35.342793 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:35.342763    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mtr5\" (UniqueName: \"kubernetes.io/projected/b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969-kube-api-access-5mtr5\") pod \"b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969\" (UID: \"b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969\") "
Sep 17 00:50:35.342918 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:35.342849    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969-extensions-socket-volume\") pod \"b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969\" (UID: \"b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969\") "
Sep 17 00:50:35.343221 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:35.343198    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969" (UID: "b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 00:50:35.345003 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:35.344971    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969-kube-api-access-5mtr5" (OuterVolumeSpecName: "kube-api-access-5mtr5") pod "b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969" (UID: "b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969"). InnerVolumeSpecName "kube-api-access-5mtr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:50:35.361133 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:35.361107    2577 generic.go:356] "Generic (PLEG): container finished" podID="b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969" containerID="ee7032d0f71608edb5109a82bab6c54e8e3fe94be0188600bfef1b87c8a63b14" exitCode=0
Sep 17 00:50:35.361251 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:35.361148    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg"
Sep 17 00:50:35.361251 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:35.361199    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg" event={"ID":"b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969","Type":"ContainerDied","Data":"ee7032d0f71608edb5109a82bab6c54e8e3fe94be0188600bfef1b87c8a63b14"}
Sep 17 00:50:35.361380 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:35.361258    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg" event={"ID":"b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969","Type":"ContainerDied","Data":"765d4a02261280ab5a36594f53ddfcd7b9190a0dbc33ed3c2b83bbed39fdb1dc"}
Sep 17 00:50:35.361380 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:35.361291    2577 scope.go:117] "RemoveContainer" containerID="ee7032d0f71608edb5109a82bab6c54e8e3fe94be0188600bfef1b87c8a63b14"
Sep 17 00:50:35.373655 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:35.373631    2577 scope.go:117] "RemoveContainer" containerID="2f0c6449ce251be029d06288ee0c364626ea75e57c1be1a3339920f930194397"
Sep 17 00:50:35.383365 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:35.383340    2577 scope.go:117] "RemoveContainer" containerID="ee7032d0f71608edb5109a82bab6c54e8e3fe94be0188600bfef1b87c8a63b14"
Sep 17 00:50:35.383744 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:50:35.383712    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee7032d0f71608edb5109a82bab6c54e8e3fe94be0188600bfef1b87c8a63b14\": container with ID starting with ee7032d0f71608edb5109a82bab6c54e8e3fe94be0188600bfef1b87c8a63b14 not found: ID does not exist" containerID="ee7032d0f71608edb5109a82bab6c54e8e3fe94be0188600bfef1b87c8a63b14"
Sep 17 00:50:35.383813 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:35.383758    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee7032d0f71608edb5109a82bab6c54e8e3fe94be0188600bfef1b87c8a63b14"} err="failed to get container status \"ee7032d0f71608edb5109a82bab6c54e8e3fe94be0188600bfef1b87c8a63b14\": rpc error: code = NotFound desc = could not find container \"ee7032d0f71608edb5109a82bab6c54e8e3fe94be0188600bfef1b87c8a63b14\": container with ID starting with ee7032d0f71608edb5109a82bab6c54e8e3fe94be0188600bfef1b87c8a63b14 not found: ID does not exist"
Sep 17 00:50:35.383813 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:35.383786    2577 scope.go:117] "RemoveContainer" containerID="2f0c6449ce251be029d06288ee0c364626ea75e57c1be1a3339920f930194397"
Sep 17 00:50:35.384079 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:50:35.384059    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f0c6449ce251be029d06288ee0c364626ea75e57c1be1a3339920f930194397\": container with ID starting with 2f0c6449ce251be029d06288ee0c364626ea75e57c1be1a3339920f930194397 not found: ID does not exist" containerID="2f0c6449ce251be029d06288ee0c364626ea75e57c1be1a3339920f930194397"
Sep 17 00:50:35.384135 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:35.384087    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0c6449ce251be029d06288ee0c364626ea75e57c1be1a3339920f930194397"} err="failed to get container status \"2f0c6449ce251be029d06288ee0c364626ea75e57c1be1a3339920f930194397\": rpc error: code = NotFound desc = could not find container \"2f0c6449ce251be029d06288ee0c364626ea75e57c1be1a3339920f930194397\": container with ID starting with 2f0c6449ce251be029d06288ee0c364626ea75e57c1be1a3339920f930194397 not found: ID does not exist"
Sep 17 00:50:35.384350 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:35.384332    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg"]
Sep 17 00:50:35.391760 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:35.391732    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-64ngg"]
Sep 17 00:50:35.443969 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:35.443943    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969-extensions-socket-volume\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:50:35.443969 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:35.443966    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mtr5\" (UniqueName: \"kubernetes.io/projected/b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969-kube-api-access-5mtr5\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:50:35.607120 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:50:35.607089    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969" path="/var/lib/kubelet/pods/b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969/volumes"
Sep 17 00:51:34.871727 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:34.871689    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gcbgs"]
Sep 17 00:51:34.872195 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:34.872032    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969" containerName="manager"
Sep 17 00:51:34.872195 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:34.872044    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969" containerName="manager"
Sep 17 00:51:34.872195 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:34.872057    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969" containerName="manager"
Sep 17 00:51:34.872195 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:34.872064    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969" containerName="manager"
Sep 17 00:51:34.872195 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:34.872075    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98a28e2c-2d11-4521-aad9-c6a5846ea441" containerName="revoke-keys"
Sep 17 00:51:34.872195 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:34.872081    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="98a28e2c-2d11-4521-aad9-c6a5846ea441" containerName="revoke-keys"
Sep 17 00:51:34.872195 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:34.872142    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="98a28e2c-2d11-4521-aad9-c6a5846ea441" containerName="revoke-keys"
Sep 17 00:51:34.872195 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:34.872150    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969" containerName="manager"
Sep 17 00:51:34.872195 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:34.872156    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9e90a8a-5e50-4d2f-8cbb-3ead9ca8b969" containerName="manager"
Sep 17 00:51:34.875083 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:34.875066    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gcbgs"
Sep 17 00:51:34.877563 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:34.877545    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-n26np\""
Sep 17 00:51:34.886727 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:34.886698    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gcbgs"]
Sep 17 00:51:34.940637 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:34.940596    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2sjq\" (UniqueName: \"kubernetes.io/projected/30af44c2-3559-468d-b0d7-b26c610298d3-kube-api-access-q2sjq\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gcbgs\" (UID: \"30af44c2-3559-468d-b0d7-b26c610298d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gcbgs"
Sep 17 00:51:34.940821 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:34.940657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30af44c2-3559-468d-b0d7-b26c610298d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gcbgs\" (UID: \"30af44c2-3559-468d-b0d7-b26c610298d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gcbgs"
Sep 17 00:51:35.041259 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:35.041206    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2sjq\" (UniqueName: \"kubernetes.io/projected/30af44c2-3559-468d-b0d7-b26c610298d3-kube-api-access-q2sjq\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gcbgs\" (UID: \"30af44c2-3559-468d-b0d7-b26c610298d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gcbgs"
Sep 17 00:51:35.041259 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:35.041267    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30af44c2-3559-468d-b0d7-b26c610298d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gcbgs\" (UID: \"30af44c2-3559-468d-b0d7-b26c610298d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gcbgs"
Sep 17 00:51:35.041653 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:35.041633    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30af44c2-3559-468d-b0d7-b26c610298d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gcbgs\" (UID: \"30af44c2-3559-468d-b0d7-b26c610298d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gcbgs"
Sep 17 00:51:35.048998 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:35.048974    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2sjq\" (UniqueName: \"kubernetes.io/projected/30af44c2-3559-468d-b0d7-b26c610298d3-kube-api-access-q2sjq\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gcbgs\" (UID: \"30af44c2-3559-468d-b0d7-b26c610298d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gcbgs"
Sep 17 00:51:35.184363 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:35.184328    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gcbgs"
Sep 17 00:51:35.465439 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:35.465397    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gcbgs"]
Sep 17 00:51:35.466677 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:51:35.466649    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30af44c2_3559_468d_b0d7_b26c610298d3.slice/crio-8701f9334a751508c46ffa52de1b062ab292fd3c7cb67c919e0bf15f6a06fd64 WatchSource:0}: Error finding container 8701f9334a751508c46ffa52de1b062ab292fd3c7cb67c919e0bf15f6a06fd64: Status 404 returned error can't find the container with id 8701f9334a751508c46ffa52de1b062ab292fd3c7cb67c919e0bf15f6a06fd64
Sep 17 00:51:35.668377 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:35.668338    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gcbgs" event={"ID":"30af44c2-3559-468d-b0d7-b26c610298d3","Type":"ContainerStarted","Data":"843b6d3e0262f29db381fed5b5e766e7979652a76e625af28a249547717e8643"}
Sep 17 00:51:35.668578 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:35.668400    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gcbgs" event={"ID":"30af44c2-3559-468d-b0d7-b26c610298d3","Type":"ContainerStarted","Data":"8701f9334a751508c46ffa52de1b062ab292fd3c7cb67c919e0bf15f6a06fd64"}
Sep 17 00:51:35.668578 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:35.668484    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gcbgs"
Sep 17 00:51:35.686726 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:35.686674    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gcbgs" podStartSLOduration=1.686660046 podStartE2EDuration="1.686660046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:51:35.686484714 +0000 UTC m=+2508.544977849" watchObservedRunningTime="2026-09-17 00:51:35.686660046 +0000 UTC m=+2508.545153181"
Sep 17 00:51:46.677062 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:46.676985    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gcbgs"
Sep 17 00:51:48.036206 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:48.036170    2577 scope.go:117] "RemoveContainer" containerID="3a276c4de88145a7f01d1d26dbf574aceb75847c93fec7d363b6fad643d5992c"
Sep 17 00:51:48.044669 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:48.044632    2577 scope.go:117] "RemoveContainer" containerID="63b4a7844c311a5525baebfec99a20348ffc76330cdbc6375b82cd8f52e705b3"
Sep 17 00:51:55.092944 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:51:55.092907    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:52:01.794170 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:52:01.794130    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:52:09.699237 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:52:09.699195    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:52:21.507034 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:52:21.506995    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:52:31.207634 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:52:31.207594    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:52:56.502170 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:52:56.502132    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:53:04.603972 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:04.603934    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:53:12.894180 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:12.894101    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:53:12.986385 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:12.986330    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc"]
Sep 17 00:53:12.993576 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:12.993532    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc"
Sep 17 00:53:12.997018 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:12.996989    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 00:53:12.997162 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:12.997058    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 17 00:53:12.997162 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:12.997086    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2nbqn\""
Sep 17 00:53:12.997162 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:12.996991    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 17 00:53:12.997162 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:12.997139    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 17 00:53:13.001288 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:13.001138    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc"]
Sep 17 00:53:13.048070 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:13.048031    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb3e505f-31ea-4e2f-996b-5978d9a413ae-maas-api-tls\") pod \"maas-api-7d747b4f9d-l9sbc\" (UID: \"cb3e505f-31ea-4e2f-996b-5978d9a413ae\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc"
Sep 17 00:53:13.048276 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:13.048159    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hblz7\" (UniqueName: \"kubernetes.io/projected/cb3e505f-31ea-4e2f-996b-5978d9a413ae-kube-api-access-hblz7\") pod \"maas-api-7d747b4f9d-l9sbc\" (UID: \"cb3e505f-31ea-4e2f-996b-5978d9a413ae\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc"
Sep 17 00:53:13.048276 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:13.048200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3e505f-31ea-4e2f-996b-5978d9a413ae-openshift-service-ca\") pod \"maas-api-7d747b4f9d-l9sbc\" (UID: \"cb3e505f-31ea-4e2f-996b-5978d9a413ae\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc"
Sep 17 00:53:13.048276 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:13.048223    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3e505f-31ea-4e2f-996b-5978d9a413ae-metrics-certs\") pod \"maas-api-7d747b4f9d-l9sbc\" (UID: \"cb3e505f-31ea-4e2f-996b-5978d9a413ae\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc"
Sep 17 00:53:13.149005 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:13.148906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hblz7\" (UniqueName: \"kubernetes.io/projected/cb3e505f-31ea-4e2f-996b-5978d9a413ae-kube-api-access-hblz7\") pod \"maas-api-7d747b4f9d-l9sbc\" (UID: \"cb3e505f-31ea-4e2f-996b-5978d9a413ae\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc"
Sep 17 00:53:13.149005 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:13.148954    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3e505f-31ea-4e2f-996b-5978d9a413ae-openshift-service-ca\") pod \"maas-api-7d747b4f9d-l9sbc\" (UID: \"cb3e505f-31ea-4e2f-996b-5978d9a413ae\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc"
Sep 17 00:53:13.149005 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:13.148980    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3e505f-31ea-4e2f-996b-5978d9a413ae-metrics-certs\") pod \"maas-api-7d747b4f9d-l9sbc\" (UID: \"cb3e505f-31ea-4e2f-996b-5978d9a413ae\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc"
Sep 17 00:53:13.149332 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:13.149033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb3e505f-31ea-4e2f-996b-5978d9a413ae-maas-api-tls\") pod \"maas-api-7d747b4f9d-l9sbc\" (UID: \"cb3e505f-31ea-4e2f-996b-5978d9a413ae\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc"
Sep 17 00:53:13.149617 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:13.149588    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3e505f-31ea-4e2f-996b-5978d9a413ae-openshift-service-ca\") pod \"maas-api-7d747b4f9d-l9sbc\" (UID: \"cb3e505f-31ea-4e2f-996b-5978d9a413ae\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc"
Sep 17 00:53:13.151480 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:13.151455    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3e505f-31ea-4e2f-996b-5978d9a413ae-metrics-certs\") pod \"maas-api-7d747b4f9d-l9sbc\" (UID: \"cb3e505f-31ea-4e2f-996b-5978d9a413ae\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc"
Sep 17 00:53:13.151586 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:13.151523    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb3e505f-31ea-4e2f-996b-5978d9a413ae-maas-api-tls\") pod \"maas-api-7d747b4f9d-l9sbc\" (UID: \"cb3e505f-31ea-4e2f-996b-5978d9a413ae\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc"
Sep 17 00:53:13.156760 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:13.156735    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hblz7\" (UniqueName: \"kubernetes.io/projected/cb3e505f-31ea-4e2f-996b-5978d9a413ae-kube-api-access-hblz7\") pod \"maas-api-7d747b4f9d-l9sbc\" (UID: \"cb3e505f-31ea-4e2f-996b-5978d9a413ae\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc"
Sep 17 00:53:13.306031 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:13.305987    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc"
Sep 17 00:53:13.581638 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:13.581608    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc"]
Sep 17 00:53:13.583535 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:53:13.583506    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3e505f_31ea_4e2f_996b_5978d9a413ae.slice/crio-d7b2dcfa112f7c9e0fac8cff8fe7cb10e71fc479d9ef1480ea5f45e8a8f5d388 WatchSource:0}: Error finding container d7b2dcfa112f7c9e0fac8cff8fe7cb10e71fc479d9ef1480ea5f45e8a8f5d388: Status 404 returned error can't find the container with id d7b2dcfa112f7c9e0fac8cff8fe7cb10e71fc479d9ef1480ea5f45e8a8f5d388
Sep 17 00:53:13.585029 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:13.585009    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 00:53:14.202280 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:14.202244    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc" event={"ID":"cb3e505f-31ea-4e2f-996b-5978d9a413ae","Type":"ContainerStarted","Data":"ce755856a4250445554cbceb40a1fa31c60aa7e78c6bd56c9fd64600abe32d05"}
Sep 17 00:53:14.202781 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:14.202303    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc" event={"ID":"cb3e505f-31ea-4e2f-996b-5978d9a413ae","Type":"ContainerStarted","Data":"d7b2dcfa112f7c9e0fac8cff8fe7cb10e71fc479d9ef1480ea5f45e8a8f5d388"}
Sep 17 00:53:14.202781 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:14.202352    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc"
Sep 17 00:53:14.219369 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:14.219321    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc" podStartSLOduration=1.813507449 podStartE2EDuration="2.219306409s" totalImagesPullingTime="405.79896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:53:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:53:14.217594369 +0000 UTC m=+2607.076087517" watchObservedRunningTime="2026-09-17 00:53:14.219306409 +0000 UTC m=+2607.077799600"
Sep 17 00:53:20.212498 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:20.212467    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc"
Sep 17 00:53:20.355520 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:20.355488    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6789c486f4-jp69r"]
Sep 17 00:53:20.358834 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:20.358817    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6789c486f4-jp69r"
Sep 17 00:53:20.366216 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:20.366193    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6789c486f4-jp69r"]
Sep 17 00:53:20.517629 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:20.517537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d69fe18-0288-44c0-9520-6bf0b40a22ed-metrics-certs\") pod \"maas-api-6789c486f4-jp69r\" (UID: \"6d69fe18-0288-44c0-9520-6bf0b40a22ed\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-jp69r"
Sep 17 00:53:20.517629 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:20.517578    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d69fe18-0288-44c0-9520-6bf0b40a22ed-openshift-service-ca\") pod \"maas-api-6789c486f4-jp69r\" (UID: \"6d69fe18-0288-44c0-9520-6bf0b40a22ed\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-jp69r"
Sep 17 00:53:20.517629 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:20.517609    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d69fe18-0288-44c0-9520-6bf0b40a22ed-maas-api-tls\") pod \"maas-api-6789c486f4-jp69r\" (UID: \"6d69fe18-0288-44c0-9520-6bf0b40a22ed\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-jp69r"
Sep 17 00:53:20.517858 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:20.517688    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tq6p\" (UniqueName: \"kubernetes.io/projected/6d69fe18-0288-44c0-9520-6bf0b40a22ed-kube-api-access-6tq6p\") pod \"maas-api-6789c486f4-jp69r\" (UID: \"6d69fe18-0288-44c0-9520-6bf0b40a22ed\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-jp69r"
Sep 17 00:53:20.618753 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:20.618716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d69fe18-0288-44c0-9520-6bf0b40a22ed-metrics-certs\") pod \"maas-api-6789c486f4-jp69r\" (UID: \"6d69fe18-0288-44c0-9520-6bf0b40a22ed\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-jp69r"
Sep 17 00:53:20.618753 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:20.618755    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d69fe18-0288-44c0-9520-6bf0b40a22ed-openshift-service-ca\") pod \"maas-api-6789c486f4-jp69r\" (UID: \"6d69fe18-0288-44c0-9520-6bf0b40a22ed\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-jp69r"
Sep 17 00:53:20.618965 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:20.618784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d69fe18-0288-44c0-9520-6bf0b40a22ed-maas-api-tls\") pod \"maas-api-6789c486f4-jp69r\" (UID: \"6d69fe18-0288-44c0-9520-6bf0b40a22ed\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-jp69r"
Sep 17 00:53:20.618965 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:20.618817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tq6p\" (UniqueName: \"kubernetes.io/projected/6d69fe18-0288-44c0-9520-6bf0b40a22ed-kube-api-access-6tq6p\") pod \"maas-api-6789c486f4-jp69r\" (UID: \"6d69fe18-0288-44c0-9520-6bf0b40a22ed\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-jp69r"
Sep 17 00:53:20.619438 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:20.619378    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d69fe18-0288-44c0-9520-6bf0b40a22ed-openshift-service-ca\") pod \"maas-api-6789c486f4-jp69r\" (UID: \"6d69fe18-0288-44c0-9520-6bf0b40a22ed\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-jp69r"
Sep 17 00:53:20.621231 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:20.621206    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d69fe18-0288-44c0-9520-6bf0b40a22ed-metrics-certs\") pod \"maas-api-6789c486f4-jp69r\" (UID: \"6d69fe18-0288-44c0-9520-6bf0b40a22ed\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-jp69r"
Sep 17 00:53:20.621368 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:20.621345    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d69fe18-0288-44c0-9520-6bf0b40a22ed-maas-api-tls\") pod \"maas-api-6789c486f4-jp69r\" (UID: \"6d69fe18-0288-44c0-9520-6bf0b40a22ed\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-jp69r"
Sep 17 00:53:20.626642 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:20.626612    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tq6p\" (UniqueName: \"kubernetes.io/projected/6d69fe18-0288-44c0-9520-6bf0b40a22ed-kube-api-access-6tq6p\") pod \"maas-api-6789c486f4-jp69r\" (UID: \"6d69fe18-0288-44c0-9520-6bf0b40a22ed\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-jp69r"
Sep 17 00:53:20.671268 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:20.671231    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6789c486f4-jp69r"
Sep 17 00:53:20.697621 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:20.697588    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:53:20.942239 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:20.942210    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6789c486f4-jp69r"]
Sep 17 00:53:20.944160 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:53:20.944127    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d69fe18_0288_44c0_9520_6bf0b40a22ed.slice/crio-0a0c121f830a5a64ccd251c19baed8a4675d21f93b87ed8fb3e97e42aa58b284 WatchSource:0}: Error finding container 0a0c121f830a5a64ccd251c19baed8a4675d21f93b87ed8fb3e97e42aa58b284: Status 404 returned error can't find the container with id 0a0c121f830a5a64ccd251c19baed8a4675d21f93b87ed8fb3e97e42aa58b284
Sep 17 00:53:21.238808 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:21.238725    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6789c486f4-jp69r" event={"ID":"6d69fe18-0288-44c0-9520-6bf0b40a22ed","Type":"ContainerStarted","Data":"0a0c121f830a5a64ccd251c19baed8a4675d21f93b87ed8fb3e97e42aa58b284"}
Sep 17 00:53:22.245740 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:22.245699    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6789c486f4-jp69r" event={"ID":"6d69fe18-0288-44c0-9520-6bf0b40a22ed","Type":"ContainerStarted","Data":"2cc93fadf891ae3573d9315d465d4eff90cb77f7c2e6c64d08cf983898b4e8ea"}
Sep 17 00:53:22.246165 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:22.245761    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6789c486f4-jp69r"
Sep 17 00:53:22.264024 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:22.263966    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6789c486f4-jp69r" podStartSLOduration=1.8704926670000002 podStartE2EDuration="2.263944718s" totalImagesPullingTime="393.452051ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:53:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:53:22.261821974 +0000 UTC m=+2615.120315109" watchObservedRunningTime="2026-09-17 00:53:22.263944718 +0000 UTC m=+2615.122437854"
Sep 17 00:53:27.704143 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:27.704104    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:53:28.256630 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:28.256596    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6789c486f4-jp69r"
Sep 17 00:53:28.303500 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:28.303466    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc"]
Sep 17 00:53:28.303747 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:28.303700    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc" podUID="cb3e505f-31ea-4e2f-996b-5978d9a413ae" containerName="maas-api" containerID="cri-o://ce755856a4250445554cbceb40a1fa31c60aa7e78c6bd56c9fd64600abe32d05" gracePeriod=30
Sep 17 00:53:29.547258 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:29.547224    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc"
Sep 17 00:53:29.696246 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:29.696215    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb3e505f-31ea-4e2f-996b-5978d9a413ae-maas-api-tls\") pod \"cb3e505f-31ea-4e2f-996b-5978d9a413ae\" (UID: \"cb3e505f-31ea-4e2f-996b-5978d9a413ae\") "
Sep 17 00:53:29.696467 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:29.696303    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3e505f-31ea-4e2f-996b-5978d9a413ae-metrics-certs\") pod \"cb3e505f-31ea-4e2f-996b-5978d9a413ae\" (UID: \"cb3e505f-31ea-4e2f-996b-5978d9a413ae\") "
Sep 17 00:53:29.696467 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:29.696339    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hblz7\" (UniqueName: \"kubernetes.io/projected/cb3e505f-31ea-4e2f-996b-5978d9a413ae-kube-api-access-hblz7\") pod \"cb3e505f-31ea-4e2f-996b-5978d9a413ae\" (UID: \"cb3e505f-31ea-4e2f-996b-5978d9a413ae\") "
Sep 17 00:53:29.696467 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:29.696451    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3e505f-31ea-4e2f-996b-5978d9a413ae-openshift-service-ca\") pod \"cb3e505f-31ea-4e2f-996b-5978d9a413ae\" (UID: \"cb3e505f-31ea-4e2f-996b-5978d9a413ae\") "
Sep 17 00:53:29.696870 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:29.696835    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3e505f-31ea-4e2f-996b-5978d9a413ae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cb3e505f-31ea-4e2f-996b-5978d9a413ae" (UID: "cb3e505f-31ea-4e2f-996b-5978d9a413ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 00:53:29.698299 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:29.698276    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb3e505f-31ea-4e2f-996b-5978d9a413ae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cb3e505f-31ea-4e2f-996b-5978d9a413ae" (UID: "cb3e505f-31ea-4e2f-996b-5978d9a413ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:53:29.698540 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:29.698519    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3e505f-31ea-4e2f-996b-5978d9a413ae-kube-api-access-hblz7" (OuterVolumeSpecName: "kube-api-access-hblz7") pod "cb3e505f-31ea-4e2f-996b-5978d9a413ae" (UID: "cb3e505f-31ea-4e2f-996b-5978d9a413ae"). InnerVolumeSpecName "kube-api-access-hblz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 00:53:29.698600 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:29.698550    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb3e505f-31ea-4e2f-996b-5978d9a413ae-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cb3e505f-31ea-4e2f-996b-5978d9a413ae" (UID: "cb3e505f-31ea-4e2f-996b-5978d9a413ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 00:53:29.797196 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:29.797164    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hblz7\" (UniqueName: \"kubernetes.io/projected/cb3e505f-31ea-4e2f-996b-5978d9a413ae-kube-api-access-hblz7\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:53:29.797196 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:29.797192    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb3e505f-31ea-4e2f-996b-5978d9a413ae-openshift-service-ca\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:53:29.797196 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:29.797203    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb3e505f-31ea-4e2f-996b-5978d9a413ae-maas-api-tls\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:53:29.797438 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:29.797212    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb3e505f-31ea-4e2f-996b-5978d9a413ae-metrics-certs\") on node \"ip-10-0-139-27.ec2.internal\" DevicePath \"\""
Sep 17 00:53:30.287827 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:30.287791    2577 generic.go:356] "Generic (PLEG): container finished" podID="cb3e505f-31ea-4e2f-996b-5978d9a413ae" containerID="ce755856a4250445554cbceb40a1fa31c60aa7e78c6bd56c9fd64600abe32d05" exitCode=0
Sep 17 00:53:30.288000 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:30.287831    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc"
Sep 17 00:53:30.288000 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:30.287844    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc" event={"ID":"cb3e505f-31ea-4e2f-996b-5978d9a413ae","Type":"ContainerDied","Data":"ce755856a4250445554cbceb40a1fa31c60aa7e78c6bd56c9fd64600abe32d05"}
Sep 17 00:53:30.288000 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:30.287894    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc" event={"ID":"cb3e505f-31ea-4e2f-996b-5978d9a413ae","Type":"ContainerDied","Data":"d7b2dcfa112f7c9e0fac8cff8fe7cb10e71fc479d9ef1480ea5f45e8a8f5d388"}
Sep 17 00:53:30.288000 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:30.287918    2577 scope.go:117] "RemoveContainer" containerID="ce755856a4250445554cbceb40a1fa31c60aa7e78c6bd56c9fd64600abe32d05"
Sep 17 00:53:30.297238 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:30.297220    2577 scope.go:117] "RemoveContainer" containerID="ce755856a4250445554cbceb40a1fa31c60aa7e78c6bd56c9fd64600abe32d05"
Sep 17 00:53:30.297498 ip-10-0-139-27 kubenswrapper[2577]: E0917 00:53:30.297479    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce755856a4250445554cbceb40a1fa31c60aa7e78c6bd56c9fd64600abe32d05\": container with ID starting with ce755856a4250445554cbceb40a1fa31c60aa7e78c6bd56c9fd64600abe32d05 not found: ID does not exist" containerID="ce755856a4250445554cbceb40a1fa31c60aa7e78c6bd56c9fd64600abe32d05"
Sep 17 00:53:30.297566 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:30.297504    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce755856a4250445554cbceb40a1fa31c60aa7e78c6bd56c9fd64600abe32d05"} err="failed to get container status \"ce755856a4250445554cbceb40a1fa31c60aa7e78c6bd56c9fd64600abe32d05\": rpc error: code = NotFound desc = could not find container \"ce755856a4250445554cbceb40a1fa31c60aa7e78c6bd56c9fd64600abe32d05\": container with ID starting with ce755856a4250445554cbceb40a1fa31c60aa7e78c6bd56c9fd64600abe32d05 not found: ID does not exist"
Sep 17 00:53:30.309188 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:30.309167    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc"]
Sep 17 00:53:30.312385 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:30.312365    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d747b4f9d-l9sbc"]
Sep 17 00:53:31.608159 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:31.608126    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb3e505f-31ea-4e2f-996b-5978d9a413ae" path="/var/lib/kubelet/pods/cb3e505f-31ea-4e2f-996b-5978d9a413ae/volumes"
Sep 17 00:53:35.994612 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:35.994576    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:53:41.306864 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:41.306829    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-r7x6w"]
Sep 17 00:53:48.073803 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:48.073770    2577 scope.go:117] "RemoveContainer" containerID="14f2caa1d1c949943d0fce807155bbc87483ada22dd78f8b6839cfd307bacb8b"
Sep 17 00:53:49.768205 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:49.768177    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-mnjsh_c85fdce2-ade0-4b20-b82d-0decac04bf20/manager/0.log"
Sep 17 00:53:49.886682 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:49.886650    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-96l8l_e37901a4-81e2-41bb-bb2a-1acf51d61759/manager/1.log"
Sep 17 00:53:50.380281 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:50.380246    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cc689bf9d-h477m_ffb945e2-7e64-4c22-a080-97685816fd25/copy-manifests/0.log"
Sep 17 00:53:50.386134 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:50.386111    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cc689bf9d-h477m_ffb945e2-7e64-4c22-a080-97685816fd25/manager/0.log"
Sep 17 00:53:50.983668 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:50.983628    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6789c486f4-jp69r_6d69fe18-0288-44c0-9520-6bf0b40a22ed/maas-api/0.log"
Sep 17 00:53:52.360283 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:52.360251    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-kqwmq_bc5b38a8-e333-4379-ad37-ddbfa0744717/manager/0.log"
Sep 17 00:53:52.596079 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:52.596050    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-f5gwq_438275c3-8268-4fb3-a4f2-0f8c75ed3b89/kuadrant-console-plugin/0.log"
Sep 17 00:53:52.720795 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:52.720762    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-gcbgs_30af44c2-3559-468d-b0d7-b26c610298d3/manager/0.log"
Sep 17 00:53:52.860409 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:52.860381    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-r7x6w_d4d3ebb4-cb8f-4a46-a9b7-deef9a612af2/limitador/0.log"
Sep 17 00:53:52.980024 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:52.979944    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-7p54z_c5c2d4fb-e8b4-478a-82b3-ade5fa66e810/manager/0.log"
Sep 17 00:53:53.594259 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:53.594231    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f57c6d865-67rlh_8305f43c-0a13-42e3-89f3-67da95cbd3bd/kube-auth-proxy/0.log"
Sep 17 00:53:54.672918 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:54.672887    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-kzq6k_b5ccd1e5-3fb1-41f4-a707-28de7a34b07e/main/0.log"
Sep 17 00:53:54.908628 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:54.908598    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-728fr_4625e67b-a3ce-4f58-86b6-c0224538815f/main/0.log"
Sep 17 00:53:55.136881 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:55.136838    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-bdhcj_d1d34169-b614-4582-954a-8f3bd7f72819/main/0.log"
Sep 17 00:53:55.251497 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:53:55.251468    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-5kdf2_176b5107-4763-404b-b51b-39bd741a6835/main/0.log"
Sep 17 00:54:03.917884 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:03.917848    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ch5w2_504ae0de-dca4-48aa-964c-b457c02267a5/global-pull-secret-syncer/0.log"
Sep 17 00:54:04.071821 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:04.071792    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-w8ps8_c956d3aa-d437-4b53-afc6-56127f4e1688/konnectivity-agent/0.log"
Sep 17 00:54:04.135481 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:04.135455    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-27.ec2.internal_fbf1e8714548f742e80d275d4df6599c/haproxy/0.log"
Sep 17 00:54:09.256338 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:09.256304    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-kqwmq_bc5b38a8-e333-4379-ad37-ddbfa0744717/manager/0.log"
Sep 17 00:54:09.303869 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:09.303841    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-f5gwq_438275c3-8268-4fb3-a4f2-0f8c75ed3b89/kuadrant-console-plugin/0.log"
Sep 17 00:54:09.338140 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:09.338110    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-gcbgs_30af44c2-3559-468d-b0d7-b26c610298d3/manager/0.log"
Sep 17 00:54:09.361318 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:09.361292    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-r7x6w_d4d3ebb4-cb8f-4a46-a9b7-deef9a612af2/limitador/0.log"
Sep 17 00:54:09.411667 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:09.411637    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-7p54z_c5c2d4fb-e8b4-478a-82b3-ade5fa66e810/manager/0.log"
Sep 17 00:54:11.238139 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:11.238111    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xj545_fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4/node-exporter/0.log"
Sep 17 00:54:11.262359 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:11.262334    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xj545_fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4/kube-rbac-proxy/0.log"
Sep 17 00:54:11.279906 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:11.279886    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xj545_fe3f2779-f3e8-4d7d-9f08-95fa98e5c6a4/init-textfile/0.log"
Sep 17 00:54:13.280520 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:13.280486    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-cch7x_b91229a2-6f57-4a69-af2c-8cd92b0459f0/networking-console-plugin/0.log"
Sep 17 00:54:14.322259 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.322222    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v"]
Sep 17 00:54:14.322683 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.322584    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb3e505f-31ea-4e2f-996b-5978d9a413ae" containerName="maas-api"
Sep 17 00:54:14.322683 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.322596    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb3e505f-31ea-4e2f-996b-5978d9a413ae" containerName="maas-api"
Sep 17 00:54:14.322683 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.322650    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb3e505f-31ea-4e2f-996b-5978d9a413ae" containerName="maas-api"
Sep 17 00:54:14.325631 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.325610    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v"
Sep 17 00:54:14.328430 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.328398    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-d74xj\"/\"default-dockercfg-l5959\""
Sep 17 00:54:14.328560 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.328398    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d74xj\"/\"openshift-service-ca.crt\""
Sep 17 00:54:14.329535 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.329514    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d74xj\"/\"kube-root-ca.crt\""
Sep 17 00:54:14.333707 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.333681    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v"]
Sep 17 00:54:14.486973 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.486934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss5sz\" (UniqueName: \"kubernetes.io/projected/e3e06f01-346a-4d4b-9dc6-5cea0dc484e5-kube-api-access-ss5sz\") pod \"perf-node-gather-daemonset-zkp6v\" (UID: \"e3e06f01-346a-4d4b-9dc6-5cea0dc484e5\") " pod="openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v"
Sep 17 00:54:14.487152 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.486987    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3e06f01-346a-4d4b-9dc6-5cea0dc484e5-sys\") pod \"perf-node-gather-daemonset-zkp6v\" (UID: \"e3e06f01-346a-4d4b-9dc6-5cea0dc484e5\") " pod="openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v"
Sep 17 00:54:14.487152 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.487050    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3e06f01-346a-4d4b-9dc6-5cea0dc484e5-lib-modules\") pod \"perf-node-gather-daemonset-zkp6v\" (UID: \"e3e06f01-346a-4d4b-9dc6-5cea0dc484e5\") " pod="openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v"
Sep 17 00:54:14.487152 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.487124    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e3e06f01-346a-4d4b-9dc6-5cea0dc484e5-proc\") pod \"perf-node-gather-daemonset-zkp6v\" (UID: \"e3e06f01-346a-4d4b-9dc6-5cea0dc484e5\") " pod="openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v"
Sep 17 00:54:14.487268 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.487155    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e3e06f01-346a-4d4b-9dc6-5cea0dc484e5-podres\") pod \"perf-node-gather-daemonset-zkp6v\" (UID: \"e3e06f01-346a-4d4b-9dc6-5cea0dc484e5\") " pod="openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v"
Sep 17 00:54:14.588177 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.588095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3e06f01-346a-4d4b-9dc6-5cea0dc484e5-sys\") pod \"perf-node-gather-daemonset-zkp6v\" (UID: \"e3e06f01-346a-4d4b-9dc6-5cea0dc484e5\") " pod="openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v"
Sep 17 00:54:14.588177 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.588130    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3e06f01-346a-4d4b-9dc6-5cea0dc484e5-lib-modules\") pod \"perf-node-gather-daemonset-zkp6v\" (UID: \"e3e06f01-346a-4d4b-9dc6-5cea0dc484e5\") " pod="openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v"
Sep 17 00:54:14.588177 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.588156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e3e06f01-346a-4d4b-9dc6-5cea0dc484e5-proc\") pod \"perf-node-gather-daemonset-zkp6v\" (UID: \"e3e06f01-346a-4d4b-9dc6-5cea0dc484e5\") " pod="openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v"
Sep 17 00:54:14.588177 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.588174    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e3e06f01-346a-4d4b-9dc6-5cea0dc484e5-podres\") pod \"perf-node-gather-daemonset-zkp6v\" (UID: \"e3e06f01-346a-4d4b-9dc6-5cea0dc484e5\") " pod="openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v"
Sep 17 00:54:14.588486 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.588218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss5sz\" (UniqueName: \"kubernetes.io/projected/e3e06f01-346a-4d4b-9dc6-5cea0dc484e5-kube-api-access-ss5sz\") pod \"perf-node-gather-daemonset-zkp6v\" (UID: \"e3e06f01-346a-4d4b-9dc6-5cea0dc484e5\") " pod="openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v"
Sep 17 00:54:14.588486 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.588221    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3e06f01-346a-4d4b-9dc6-5cea0dc484e5-sys\") pod \"perf-node-gather-daemonset-zkp6v\" (UID: \"e3e06f01-346a-4d4b-9dc6-5cea0dc484e5\") " pod="openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v"
Sep 17 00:54:14.588486 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.588256    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e3e06f01-346a-4d4b-9dc6-5cea0dc484e5-proc\") pod \"perf-node-gather-daemonset-zkp6v\" (UID: \"e3e06f01-346a-4d4b-9dc6-5cea0dc484e5\") " pod="openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v"
Sep 17 00:54:14.588486 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.588289    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3e06f01-346a-4d4b-9dc6-5cea0dc484e5-lib-modules\") pod \"perf-node-gather-daemonset-zkp6v\" (UID: \"e3e06f01-346a-4d4b-9dc6-5cea0dc484e5\") " pod="openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v"
Sep 17 00:54:14.588486 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.588326    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e3e06f01-346a-4d4b-9dc6-5cea0dc484e5-podres\") pod \"perf-node-gather-daemonset-zkp6v\" (UID: \"e3e06f01-346a-4d4b-9dc6-5cea0dc484e5\") " pod="openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v"
Sep 17 00:54:14.595769 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.595741    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss5sz\" (UniqueName: \"kubernetes.io/projected/e3e06f01-346a-4d4b-9dc6-5cea0dc484e5-kube-api-access-ss5sz\") pod \"perf-node-gather-daemonset-zkp6v\" (UID: \"e3e06f01-346a-4d4b-9dc6-5cea0dc484e5\") " pod="openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v"
Sep 17 00:54:14.636973 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.636942    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v"
Sep 17 00:54:14.908265 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:14.908226    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v"]
Sep 17 00:54:14.910323 ip-10-0-139-27 kubenswrapper[2577]: W0917 00:54:14.910294    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode3e06f01_346a_4d4b_9dc6_5cea0dc484e5.slice/crio-b7662182661a3f94215a2eca295e0065739ac3fa5a4cd4c47b4191d0bb2f07e5 WatchSource:0}: Error finding container b7662182661a3f94215a2eca295e0065739ac3fa5a4cd4c47b4191d0bb2f07e5: Status 404 returned error can't find the container with id b7662182661a3f94215a2eca295e0065739ac3fa5a4cd4c47b4191d0bb2f07e5
Sep 17 00:54:15.534601 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:15.534565    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v" event={"ID":"e3e06f01-346a-4d4b-9dc6-5cea0dc484e5","Type":"ContainerStarted","Data":"bf362b0ba71184a278f8d500db8339a55599136a5a3001f6842d812e0c1090f5"}
Sep 17 00:54:15.535014 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:15.534638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v" event={"ID":"e3e06f01-346a-4d4b-9dc6-5cea0dc484e5","Type":"ContainerStarted","Data":"b7662182661a3f94215a2eca295e0065739ac3fa5a4cd4c47b4191d0bb2f07e5"}
Sep 17 00:54:15.535014 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:15.534717    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v"
Sep 17 00:54:15.555640 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:15.555592    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v" podStartSLOduration=1.5555773739999998 podStartE2EDuration="1.555577374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:54:15.553488778 +0000 UTC m=+2668.411981913" watchObservedRunningTime="2026-09-17 00:54:15.555577374 +0000 UTC m=+2668.414070509"
Sep 17 00:54:16.352581 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:16.352548    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zhxfj_57f31c58-e354-4d56-8b7e-c55015c26176/dns/0.log"
Sep 17 00:54:16.370610 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:16.370585    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zhxfj_57f31c58-e354-4d56-8b7e-c55015c26176/kube-rbac-proxy/0.log"
Sep 17 00:54:16.415474 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:16.415448    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9xxfq_3e97bfc2-d46a-4384-bc1a-a1a2d37afe75/dns-node-resolver/0.log"
Sep 17 00:54:17.011754 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:17.011723    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rg8zr_e6570468-fe3a-4512-8bda-9880edc3ff96/node-ca/0.log"
Sep 17 00:54:18.821008 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:18.820980    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f57c6d865-67rlh_8305f43c-0a13-42e3-89f3-67da95cbd3bd/kube-auth-proxy/0.log"
Sep 17 00:54:19.891125 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:19.891081    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-twmbk_ccc9a7ce-9b6a-40f4-9d5e-388c3e892f11/serve-healthcheck-canary/0.log"
Sep 17 00:54:20.507677 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:20.507634    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7rtx9_8d564ffa-7392-4e02-b8fb-34c857ae3874/kube-rbac-proxy/0.log"
Sep 17 00:54:20.525427 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:20.525393    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7rtx9_8d564ffa-7392-4e02-b8fb-34c857ae3874/exporter/0.log"
Sep 17 00:54:20.543293 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:20.543271    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7rtx9_8d564ffa-7392-4e02-b8fb-34c857ae3874/extractor/0.log"
Sep 17 00:54:21.548637 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:21.548607    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-d74xj/perf-node-gather-daemonset-zkp6v"
Sep 17 00:54:22.534763 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:22.534734    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-mnjsh_c85fdce2-ade0-4b20-b82d-0decac04bf20/manager/0.log"
Sep 17 00:54:22.555315 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:22.555284    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-96l8l_e37901a4-81e2-41bb-bb2a-1acf51d61759/manager/0.log"
Sep 17 00:54:22.776235 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:22.776203    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-96l8l_e37901a4-81e2-41bb-bb2a-1acf51d61759/manager/1.log"
Sep 17 00:54:23.020585 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:23.020556    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cc689bf9d-h477m_ffb945e2-7e64-4c22-a080-97685816fd25/manager/0.log"
Sep 17 00:54:23.037644 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:23.037618    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cc689bf9d-h477m_ffb945e2-7e64-4c22-a080-97685816fd25/copy-manifests/0.log"
Sep 17 00:54:24.511054 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:24.511026    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-vdppl_a6ea1ee9-37a6-40e6-bb74-a58075f729d5/openshift-lws-operator/0.log"
Sep 17 00:54:25.126848 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:25.124351    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f57c6d865-67rlh_8305f43c-0a13-42e3-89f3-67da95cbd3bd/kube-auth-proxy/0.log"
Sep 17 00:54:27.640598 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:27.640571    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f57c6d865-67rlh_8305f43c-0a13-42e3-89f3-67da95cbd3bd/kube-auth-proxy/0.log"
Sep 17 00:54:31.223457 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:31.223430    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g4cmx_30b22991-ec04-4b85-9b9d-87959cfa30d1/kube-multus-additional-cni-plugins/0.log"
Sep 17 00:54:31.245221 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:31.245191    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g4cmx_30b22991-ec04-4b85-9b9d-87959cfa30d1/egress-router-binary-copy/0.log"
Sep 17 00:54:31.263712 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:31.263685    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g4cmx_30b22991-ec04-4b85-9b9d-87959cfa30d1/cni-plugins/0.log"
Sep 17 00:54:31.281280 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:31.281252    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g4cmx_30b22991-ec04-4b85-9b9d-87959cfa30d1/bond-cni-plugin/0.log"
Sep 17 00:54:31.298854 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:31.298825    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g4cmx_30b22991-ec04-4b85-9b9d-87959cfa30d1/routeoverride-cni/0.log"
Sep 17 00:54:31.319531 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:31.319506    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g4cmx_30b22991-ec04-4b85-9b9d-87959cfa30d1/whereabouts-cni-bincopy/0.log"
Sep 17 00:54:31.337176 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:31.337151    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g4cmx_30b22991-ec04-4b85-9b9d-87959cfa30d1/whereabouts-cni/0.log"
Sep 17 00:54:31.430696 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:31.430662    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-k2t7f_0a948860-0f14-46ec-afab-e23492108eaa/kube-multus/0.log"
Sep 17 00:54:31.656614 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:31.656532    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xbhz5_b956a62e-c4ac-4e1d-aeb0-3f7907a0a702/network-metrics-daemon/0.log"
Sep 17 00:54:31.672616 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:31.672584    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xbhz5_b956a62e-c4ac-4e1d-aeb0-3f7907a0a702/kube-rbac-proxy/0.log"
Sep 17 00:54:32.714462 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:32.714437    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/ovn-controller/0.log"
Sep 17 00:54:32.727145 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:32.727121    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/ovn-acl-logging/1.log"
Sep 17 00:54:32.727287 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:32.727155    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/ovn-acl-logging/0.log"
Sep 17 00:54:32.743299 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:32.743273    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/kube-rbac-proxy-node/0.log"
Sep 17 00:54:32.761029 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:32.761007    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 00:54:32.775500 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:32.775479    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/northd/0.log"
Sep 17 00:54:32.792844 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:32.792823    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/nbdb/0.log"
Sep 17 00:54:32.810354 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:32.810324    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/sbdb/0.log"
Sep 17 00:54:32.863249 ip-10-0-139-27 kubenswrapper[2577]: I0917 00:54:32.863222    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bz685_d261c1cf-76d0-468c-a122-1c2b359f3765/ovnkube-controller/0.log"